SlideShare a Scribd company logo
1 of 28
Download to read offline
Advertising Quality Science
Mounia Lalmas
This talk
4-year effort across research, engineering and product at Yahoo to
measure the quality of ads served on Gemini, Yahoo native advertising
network
Not just measuring but taking actions to improve user experience as well
as providing feedbacks to advertisers
à no deep learning but large scale predictive analytics
Focus of the talk: the post-click experience on native ads
à the quality of the landing page… if it is seen
3
The advertising world is fun J
Online advertising is big business
Values in $billions
Advertising is how Yahoo (and many other Internet companies) makes
money… and what keeps Yahoo services free for its customers
90% of Yahoo’s revenue is from advertising:
2016 search advertising revenue – $2.67B (52% of total revenue)
2016 display including native advertising – $1.98B (38% of total)
Scale: billion ads served daily
(Source: Yahoo 2016 10k annual report)
Online advertising is about connecting supply & demand
Search
Native
Display
Video
Brand
Direct
Response
Yahoo own &
operated sites
Publisher Partners
SUPPLY
(publishers)
DEMAND
(advertisers)
Advertising (ad) quality science
Develop predictive models that characterise the quality of ads shown to and clicked by users.
Maximise revenue and guaranteeing ROI to advertisers without
negatively impacting user experience.
Publishers
Advertisers
Users
Ad
inventory
ad network
Being able to help
advertisers improve the
quality of their ads
Ad Quality: Scope
Non-intentional
Ad quality
Intentional
Ad compliance
Major shift in how users access the Internet
comScore	2015		 UK Internet users
Native advertising
(Source:	Sharethrough.com	&	IPG	Media	Lab	Study:	Na;ve	Adver;sement	Effec;veness)	
Visually Engaging
Higher user attention
Higher brand lift
Social sharing
The quality of the post-click experience
the quality of the landing page on mobile
The post-click experience: Dwell time
dwell time
proxy of post-click ad
quality experience
predictive post-click ad
quality models
ad quality ratings &
recommendations
accidental clicks
identification
proxy of accidental
clicks
Metric
Ad serving
User
engagement
Advertiser
ROI
dwell time
proxy of post-click ad
quality experience
predictive post-click ad
quality models
ad quality ratings &
recommendations
accidental clicks
identification
proxy of accidental
clicks
Metric
Ad serving
User
engagement
Advertiser
ROI
The post-click experience journey
Quality of the post-click experience
Best experience is when conversion happens
No conversion does not mean a bad experience
Proxy metric of post-click experience: dwell time on the ad landing page
tad-click tback-to-publisher
dwell time = tback-to-publisher – tad-click
Positive post-click experience (“long” clicks)
has an effect on users clicking on ads again
dwell time
proxy of post-click ad
quality experience
predictive post-click ad
quality models
ad quality ratings &
recommendations
accidental clicks
identification
proxy of accidental
clicks
Metric
Ad serving
User
engagement
Advertiser
ROI
The post-click experience journey
Optimise for high quality ads
Estimating P(hq|click) = quality score
P(dwell time > t)
Build predictive models that predict if an ad is of high quality
= predicted dwell time above a given threshold t
➔  high quality = high dwell time
revenue = 𝓕 (bid, CTR, quality)
P(hq|click)
logistic regression, gradient descent boosting, random forest, survival random forest
Landing page features
●  window_size
●  view_port
●  media_support
content
mobile
support
requested
information
multimedia
mobile
optimized
out-going
connectivity
interactivity
textual
content
in-coming
connectivity
●  description
●  keywords
●  title
meta
information
●  num_forms
●  num_input_radio
●  num_input_string
●  ...
readability multimedia
significant effect of text readability and page structure
A/B testing
dwell time increased by 20%
bounce rate decreased by 7%
revenue = bid x CTR x quality
(Lalmas etal, 2015; Barbieri, Silvestri & Lalmas, 2016)
dwell time
proxy of post-click ad
quality experience
predictive post-click ad
quality models
ad quality ratings &
recommendations
accidental clicks
identification
proxy of accidental
clicks
Metric
Ad serving
User
engagement
Advertiser
ROI
The post-click experience journey
Landing page rating: Low, Average or High
landing
pages quality score q
…
L H
L and H are customisable:
e.g., LOW=[0,25%), AVG=[25%,75%], HIGH=(75%,100%]
2 cut-off points (L, H) that
divide distribution of quality
scores q into 3 regions:
-  LOW: q < L
-  AVG: L <= q <= H
-  HIGH: q > H
(L, H)
ad ratingq LOW
Improving landing pages
Exploiting the features for recommending improvements
mobile
optimized
out-going
connectivity
interactivity textual
content
in-coming
connectivity
meta
information
readability
multimedia
●  num_forms
●  num_input_radio
●  num_input_string
●  ...
interactivity
●  mediannum_forms ±ε
●  mediannum_input_radio ±ε
●  mediannum_input_string ±ε
●  ...
for each feature
compute median and
confidence interval
for each ad feature
compute the distance from
the confidence interval
given an ad
num_input_radio
num_forms
num_input_string
...
There might be too few/much textual content
There might be too few/many entities
There might be too few/many images
Landing	Page	Content	
n.	of	words	
n.	of	Wikipedia	en;;es	
n.	of	images	
Landing	Page	Layout	
height/width	
resizability	(fit	to	mul;ple	screen	size)	
Landing	Page	Structure	
n.	of	drop-down	menus	
n.	of	checkboxes	
n.	of	input	strings	
Landing	Page	Readability	
content	summarizability	
The height/width of the landing page might be too small/large
The landing page might not be adapted to different screen
sizes
There might be too many drop-down menus
There might be too many checkboxes
There might be too much information requested from the
users
The textual content might be further summarised to make it more
readable
Examples of recommendations
dwell time
proxy of post-click ad
quality experience
predictive post-click ad
quality models
ad quality ratings &
recommendations
accidental clicks
identification
proxy of accidental
clicks
Metric
Ad serving
User
engagement
Advertiser
ROI
The post-click experience journey
peak on app X
●  accidental clicks do not
reflect post-click
experience
●  not all clicks are equal
app X
The quality of a click on mobile apps
peak on app Y
dwell time distribution of apps X and Y for a
given ad
app Y
dwell time
proxy of post-click ad
quality experience
predictive post-click ad
quality models
ad quality ratings &
recommendations
accidental clicks
identification
proxy of accidental
clicks
Metric
Ad serving
User
engagement
Advertiser
ROI
The post-click experience journey
Fitting the data into a mixture model
The number of mixture components is determined using the BIC criterion
which selects the model that fits best the data while avoiding overfitting
Time period 1
Time period 2 (after UI change)
Bayesian information criterion (BIC)
bouncy clicks
accidental clicks
Accidental clicks threshold for app X
Min
1st Quartile
Median
Mean
3rd Quartile
Max
Distribution of the medians as computed on the first
component of each ad
Applications
-  discount accidental clicks
using economics models
-  train click models
discarding accident clicks
-  input to UI design
ads with all three components
dwell time
proxy of post-click ad
quality experience
predictive post-click ad
quality models
ad quality ratings &
recommendations
accidental clicks
identification
proxy of accidental
clicks
Metric
Ad serving
User
engagement
Advertiser
ROI
Ad quality: The post-click experience journey
Acknowledgments: Marc Bron, Ayman Farahat, Andy Haines, Miriam Redi, Gabriele Tolomei, Guy Shaked,
Ke (Adam) Zhou, Fabrizio Silvestri, Michele Trevisiol, Ben Shahshahani, Puneet M Sangal and many others

More Related Content

What's hot

Tips & tricks how to maximise your Facebook campaign by Andrei Ursuleanu @ Al...
Tips & tricks how to maximise your Facebook campaign by Andrei Ursuleanu @ Al...Tips & tricks how to maximise your Facebook campaign by Andrei Ursuleanu @ Al...
Tips & tricks how to maximise your Facebook campaign by Andrei Ursuleanu @ Al...ICEEFEST2013
 
Behavioral targeting
Behavioral targetingBehavioral targeting
Behavioral targetingAnil Batra
 
Digital advertising 101
Digital advertising 101Digital advertising 101
Digital advertising 101451 Marketing
 
Facebook Targeting: User Acquisition
Facebook Targeting: User AcquisitionFacebook Targeting: User Acquisition
Facebook Targeting: User AcquisitionNanigans
 
2015 Mobile Advertising Seminar
2015 Mobile Advertising Seminar2015 Mobile Advertising Seminar
2015 Mobile Advertising SeminarBenjamin Page
 
2012 Online marketing-cmis542
2012 Online marketing-cmis5422012 Online marketing-cmis542
2012 Online marketing-cmis542Pinny
 
Emerging Paid Search Platforms - Mobile & Facebook
Emerging Paid Search Platforms - Mobile & Facebook Emerging Paid Search Platforms - Mobile & Facebook
Emerging Paid Search Platforms - Mobile & Facebook Performics.Convonix
 
gemiusDirectEffect
gemiusDirectEffectgemiusDirectEffect
gemiusDirectEffectVictor Avram
 
Digital Marketing Strategy - Canvas for Bruxelles formation training
Digital Marketing Strategy - Canvas for Bruxelles formation trainingDigital Marketing Strategy - Canvas for Bruxelles formation training
Digital Marketing Strategy - Canvas for Bruxelles formation trainingDavid Hachez
 
Advertising on Google
Advertising on GoogleAdvertising on Google
Advertising on Googlemediaant
 
Nutricharge Digital Plan - OTC Medicine
Nutricharge Digital Plan - OTC MedicineNutricharge Digital Plan - OTC Medicine
Nutricharge Digital Plan - OTC Medicinemediaant
 
Two and a half men_MDI_CANVAS1
Two and a half men_MDI_CANVAS1Two and a half men_MDI_CANVAS1
Two and a half men_MDI_CANVAS1Dipesh Devani
 
Geo-Targeting and Mobile Devices
Geo-Targeting and Mobile DevicesGeo-Targeting and Mobile Devices
Geo-Targeting and Mobile DevicesSamantha4786
 
Dma detroit larry freed mar 3 2011
Dma detroit  larry freed mar 3 2011Dma detroit  larry freed mar 3 2011
Dma detroit larry freed mar 3 2011dmadetroit
 
Facebook targeting confluence con damon gochneaur
Facebook targeting confluence con damon gochneaurFacebook targeting confluence con damon gochneaur
Facebook targeting confluence con damon gochneaurDamon Gochneaur
 
Beacon Pro 360 beacons geof_brand_v1
Beacon Pro 360 beacons geof_brand_v1Beacon Pro 360 beacons geof_brand_v1
Beacon Pro 360 beacons geof_brand_v1beaconpro360
 
measuring the effectiveness of interactive media
measuring the effectiveness of interactive mediameasuring the effectiveness of interactive media
measuring the effectiveness of interactive mediapaul baker
 
The Three C’s of the Addressable Customer Experience
The Three C’s of the Addressable Customer Experience The Three C’s of the Addressable Customer Experience
The Three C’s of the Addressable Customer Experience Merkle
 
Efficient mechanisms for mobile ads
Efficient mechanisms for mobile adsEfficient mechanisms for mobile ads
Efficient mechanisms for mobile adsSandip Jalan
 

What's hot (20)

Tips & tricks how to maximise your Facebook campaign by Andrei Ursuleanu @ Al...
Tips & tricks how to maximise your Facebook campaign by Andrei Ursuleanu @ Al...Tips & tricks how to maximise your Facebook campaign by Andrei Ursuleanu @ Al...
Tips & tricks how to maximise your Facebook campaign by Andrei Ursuleanu @ Al...
 
Behavioral targeting
Behavioral targetingBehavioral targeting
Behavioral targeting
 
Digital advertising 101
Digital advertising 101Digital advertising 101
Digital advertising 101
 
Facebook Targeting: User Acquisition
Facebook Targeting: User AcquisitionFacebook Targeting: User Acquisition
Facebook Targeting: User Acquisition
 
Display Advertising for Demand Marketers
Display Advertising for Demand MarketersDisplay Advertising for Demand Marketers
Display Advertising for Demand Marketers
 
2015 Mobile Advertising Seminar
2015 Mobile Advertising Seminar2015 Mobile Advertising Seminar
2015 Mobile Advertising Seminar
 
2012 Online marketing-cmis542
2012 Online marketing-cmis5422012 Online marketing-cmis542
2012 Online marketing-cmis542
 
Emerging Paid Search Platforms - Mobile & Facebook
Emerging Paid Search Platforms - Mobile & Facebook Emerging Paid Search Platforms - Mobile & Facebook
Emerging Paid Search Platforms - Mobile & Facebook
 
gemiusDirectEffect
gemiusDirectEffectgemiusDirectEffect
gemiusDirectEffect
 
Digital Marketing Strategy - Canvas for Bruxelles formation training
Digital Marketing Strategy - Canvas for Bruxelles formation trainingDigital Marketing Strategy - Canvas for Bruxelles formation training
Digital Marketing Strategy - Canvas for Bruxelles formation training
 
Advertising on Google
Advertising on GoogleAdvertising on Google
Advertising on Google
 
Nutricharge Digital Plan - OTC Medicine
Nutricharge Digital Plan - OTC MedicineNutricharge Digital Plan - OTC Medicine
Nutricharge Digital Plan - OTC Medicine
 
Two and a half men_MDI_CANVAS1
Two and a half men_MDI_CANVAS1Two and a half men_MDI_CANVAS1
Two and a half men_MDI_CANVAS1
 
Geo-Targeting and Mobile Devices
Geo-Targeting and Mobile DevicesGeo-Targeting and Mobile Devices
Geo-Targeting and Mobile Devices
 
Dma detroit larry freed mar 3 2011
Dma detroit  larry freed mar 3 2011Dma detroit  larry freed mar 3 2011
Dma detroit larry freed mar 3 2011
 
Facebook targeting confluence con damon gochneaur
Facebook targeting confluence con damon gochneaurFacebook targeting confluence con damon gochneaur
Facebook targeting confluence con damon gochneaur
 
Beacon Pro 360 beacons geof_brand_v1
Beacon Pro 360 beacons geof_brand_v1Beacon Pro 360 beacons geof_brand_v1
Beacon Pro 360 beacons geof_brand_v1
 
measuring the effectiveness of interactive media
measuring the effectiveness of interactive mediameasuring the effectiveness of interactive media
measuring the effectiveness of interactive media
 
The Three C’s of the Addressable Customer Experience
The Three C’s of the Addressable Customer Experience The Three C’s of the Addressable Customer Experience
The Three C’s of the Addressable Customer Experience
 
Efficient mechanisms for mobile ads
Efficient mechanisms for mobile adsEfficient mechanisms for mobile ads
Efficient mechanisms for mobile ads
 

Similar to Advertising Quality Science

Introduction to Alenty
Introduction to AlentyIntroduction to Alenty
Introduction to Alentyemerceron
 
Alenty Light Master Presentation
Alenty Light   Master PresentationAlenty Light   Master Presentation
Alenty Light Master Presentationemerceron
 
How to better measure and optimize display media
How to better measure and optimize display mediaHow to better measure and optimize display media
How to better measure and optimize display mediaDavid Lee
 
Concern Digital Training 26th August 2015
Concern Digital Training 26th August 2015Concern Digital Training 26th August 2015
Concern Digital Training 26th August 2015Vanessa Vallejo
 
eCMO 2010 How to better measure and optimize display media
eCMO 2010 How to better measure and optimize display mediaeCMO 2010 How to better measure and optimize display media
eCMO 2010 How to better measure and optimize display mediaHKAIM
 
Metrics of digital marketing
Metrics of digital marketingMetrics of digital marketing
Metrics of digital marketingFriday Explorer
 
Metrics of digital marketing
Metrics of digital marketingMetrics of digital marketing
Metrics of digital marketingFriday Explorer
 
KREATIO-WHITE-PAPER----AD-REVENUE.PDF
KREATIO-WHITE-PAPER----AD-REVENUE.PDFKREATIO-WHITE-PAPER----AD-REVENUE.PDF
KREATIO-WHITE-PAPER----AD-REVENUE.PDFKadam Vivek
 
Shari Gunn
Shari GunnShari Gunn
Shari Gunntieadmin
 
Integral Ad Science Viewability Presentation
Integral Ad Science Viewability PresentationIntegral Ad Science Viewability Presentation
Integral Ad Science Viewability PresentationIntegral Ad Science
 
Discussion on affiliate marketing affiliate perspective
Discussion on affiliate marketing affiliate perspectiveDiscussion on affiliate marketing affiliate perspective
Discussion on affiliate marketing affiliate perspectiveVeeraj Vashishtha
 
Viewable Impressions Metric As The Industry Standard
Viewable Impressions Metric As The Industry StandardViewable Impressions Metric As The Industry Standard
Viewable Impressions Metric As The Industry StandardMARC USA
 
A Brief of Google AdWords Pay Per Click
A Brief of Google AdWords Pay Per ClickA Brief of Google AdWords Pay Per Click
A Brief of Google AdWords Pay Per ClickRanjan Jena
 
Display & rich media planning
Display & rich media planningDisplay & rich media planning
Display & rich media planningashish22in
 
Digital advertising- overview and study
Digital advertising- overview and studyDigital advertising- overview and study
Digital advertising- overview and studyRobin Goel
 
Campaign Digital Experiences Your Visitors Will Love
Campaign Digital Experiences Your Visitors Will LoveCampaign Digital Experiences Your Visitors Will Love
Campaign Digital Experiences Your Visitors Will Loveion interactive
 

Similar to Advertising Quality Science (20)

Alenty
AlentyAlenty
Alenty
 
Alenty
AlentyAlenty
Alenty
 
Alenty
AlentyAlenty
Alenty
 
Introduction to Alenty
Introduction to AlentyIntroduction to Alenty
Introduction to Alenty
 
Alenty Light Master Presentation
Alenty Light   Master PresentationAlenty Light   Master Presentation
Alenty Light Master Presentation
 
How to better measure and optimize display media
How to better measure and optimize display mediaHow to better measure and optimize display media
How to better measure and optimize display media
 
Concern Digital Training 26th August 2015
Concern Digital Training 26th August 2015Concern Digital Training 26th August 2015
Concern Digital Training 26th August 2015
 
eCMO 2010 How to better measure and optimize display media
eCMO 2010 How to better measure and optimize display mediaeCMO 2010 How to better measure and optimize display media
eCMO 2010 How to better measure and optimize display media
 
Alenty
AlentyAlenty
Alenty
 
Metrics of digital marketing
Metrics of digital marketingMetrics of digital marketing
Metrics of digital marketing
 
Metrics of digital marketing
Metrics of digital marketingMetrics of digital marketing
Metrics of digital marketing
 
KREATIO-WHITE-PAPER----AD-REVENUE.PDF
KREATIO-WHITE-PAPER----AD-REVENUE.PDFKREATIO-WHITE-PAPER----AD-REVENUE.PDF
KREATIO-WHITE-PAPER----AD-REVENUE.PDF
 
Shari Gunn
Shari GunnShari Gunn
Shari Gunn
 
Integral Ad Science Viewability Presentation
Integral Ad Science Viewability PresentationIntegral Ad Science Viewability Presentation
Integral Ad Science Viewability Presentation
 
Discussion on affiliate marketing affiliate perspective
Discussion on affiliate marketing affiliate perspectiveDiscussion on affiliate marketing affiliate perspective
Discussion on affiliate marketing affiliate perspective
 
Viewable Impressions Metric As The Industry Standard
Viewable Impressions Metric As The Industry StandardViewable Impressions Metric As The Industry Standard
Viewable Impressions Metric As The Industry Standard
 
A Brief of Google AdWords Pay Per Click
A Brief of Google AdWords Pay Per ClickA Brief of Google AdWords Pay Per Click
A Brief of Google AdWords Pay Per Click
 
Display & rich media planning
Display & rich media planningDisplay & rich media planning
Display & rich media planning
 
Digital advertising- overview and study
Digital advertising- overview and studyDigital advertising- overview and study
Digital advertising- overview and study
 
Campaign Digital Experiences Your Visitors Will Love
Campaign Digital Experiences Your Visitors Will LoveCampaign Digital Experiences Your Visitors Will Love
Campaign Digital Experiences Your Visitors Will Love
 

More from Mounia Lalmas-Roelleke

Engagement, Metrics & Personalisation at Scale
Engagement, Metrics &  Personalisation at ScaleEngagement, Metrics &  Personalisation at Scale
Engagement, Metrics & Personalisation at ScaleMounia Lalmas-Roelleke
 
Engagement, metrics and "recommenders"
Engagement, metrics and "recommenders"Engagement, metrics and "recommenders"
Engagement, metrics and "recommenders"Mounia Lalmas-Roelleke
 
Metrics, Engagement & Personalization
Metrics, Engagement & Personalization Metrics, Engagement & Personalization
Metrics, Engagement & Personalization Mounia Lalmas-Roelleke
 
Tutorial on Online User Engagement: Metrics and Optimization
Tutorial on Online User Engagement: Metrics and OptimizationTutorial on Online User Engagement: Metrics and Optimization
Tutorial on Online User Engagement: Metrics and OptimizationMounia Lalmas-Roelleke
 
Personalizing the listening experience
Personalizing the listening experiencePersonalizing the listening experience
Personalizing the listening experienceMounia Lalmas-Roelleke
 
Recommending and Searching (Research @ Spotify)
Recommending and Searching (Research @ Spotify)Recommending and Searching (Research @ Spotify)
Recommending and Searching (Research @ Spotify)Mounia Lalmas-Roelleke
 
An introduction to system-oriented evaluation in Information Retrieval
An introduction to system-oriented evaluation in Information RetrievalAn introduction to system-oriented evaluation in Information Retrieval
An introduction to system-oriented evaluation in Information RetrievalMounia Lalmas-Roelleke
 
Social Media and AI: Don’t forget the users
Social Media and AI: Don’t forget the usersSocial Media and AI: Don’t forget the users
Social Media and AI: Don’t forget the usersMounia Lalmas-Roelleke
 
Describing Patterns and Disruptions in Large Scale Mobile App Usage Data
Describing Patterns and Disruptions in Large Scale Mobile App Usage DataDescribing Patterns and Disruptions in Large Scale Mobile App Usage Data
Describing Patterns and Disruptions in Large Scale Mobile App Usage DataMounia Lalmas-Roelleke
 
Evaluating the search experience: from Retrieval Effectiveness to User Engage...
Evaluating the search experience: from Retrieval Effectiveness to User Engage...Evaluating the search experience: from Retrieval Effectiveness to User Engage...
Evaluating the search experience: from Retrieval Effectiveness to User Engage...Mounia Lalmas-Roelleke
 
A Journey into Evaluation: from Retrieval Effectiveness to User Engagement
A Journey into Evaluation: from Retrieval Effectiveness to User EngagementA Journey into Evaluation: from Retrieval Effectiveness to User Engagement
A Journey into Evaluation: from Retrieval Effectiveness to User EngagementMounia Lalmas-Roelleke
 
From “Selena Gomez” to “Marlon Brando”: Understanding Explorative Entity Search
From “Selena Gomez” to “Marlon Brando”: Understanding Explorative Entity SearchFrom “Selena Gomez” to “Marlon Brando”: Understanding Explorative Entity Search
From “Selena Gomez” to “Marlon Brando”: Understanding Explorative Entity SearchMounia Lalmas-Roelleke
 
How Big Data is Changing User Engagement
How Big Data is Changing User EngagementHow Big Data is Changing User Engagement
How Big Data is Changing User EngagementMounia Lalmas-Roelleke
 
Measuring user engagement: the do, the do not do, and the we do not know
Measuring user engagement: the do, the do not do, and the we do not knowMeasuring user engagement: the do, the do not do, and the we do not know
Measuring user engagement: the do, the do not do, and the we do not knowMounia Lalmas-Roelleke
 
An Engaging Click ... or how can user engagement measurement inform web searc...
An Engaging Click ... or how can user engagement measurement inform web searc...An Engaging Click ... or how can user engagement measurement inform web searc...
An Engaging Click ... or how can user engagement measurement inform web searc...Mounia Lalmas-Roelleke
 
Social Media News Communities: Gatekeeping, Coverage, and Statement Bias
 Social Media News Communities: Gatekeeping, Coverage, and Statement Bias Social Media News Communities: Gatekeeping, Coverage, and Statement Bias
Social Media News Communities: Gatekeeping, Coverage, and Statement BiasMounia Lalmas-Roelleke
 
On the Reliability and Intuitiveness of Aggregated Search Metrics
On the Reliability and Intuitiveness of Aggregated Search MetricsOn the Reliability and Intuitiveness of Aggregated Search Metrics
On the Reliability and Intuitiveness of Aggregated Search MetricsMounia Lalmas-Roelleke
 
Penguins in Sweaters, or Serendipitous Entity Search on User-generated Content
 Penguins in Sweaters, or Serendipitous Entity Search on User-generated Content Penguins in Sweaters, or Serendipitous Entity Search on User-generated Content
Penguins in Sweaters, or Serendipitous Entity Search on User-generated ContentMounia Lalmas-Roelleke
 

More from Mounia Lalmas-Roelleke (20)

Engagement, Metrics & Personalisation at Scale
Engagement, Metrics &  Personalisation at ScaleEngagement, Metrics &  Personalisation at Scale
Engagement, Metrics & Personalisation at Scale
 
Engagement, metrics and "recommenders"
Engagement, metrics and "recommenders"Engagement, metrics and "recommenders"
Engagement, metrics and "recommenders"
 
Metrics, Engagement & Personalization
Metrics, Engagement & Personalization Metrics, Engagement & Personalization
Metrics, Engagement & Personalization
 
Tutorial on Online User Engagement: Metrics and Optimization
Tutorial on Online User Engagement: Metrics and OptimizationTutorial on Online User Engagement: Metrics and Optimization
Tutorial on Online User Engagement: Metrics and Optimization
 
Recommending and searching @ Spotify
Recommending and searching @ SpotifyRecommending and searching @ Spotify
Recommending and searching @ Spotify
 
Personalizing the listening experience
Personalizing the listening experiencePersonalizing the listening experience
Personalizing the listening experience
 
Recommending and Searching (Research @ Spotify)
Recommending and Searching (Research @ Spotify)Recommending and Searching (Research @ Spotify)
Recommending and Searching (Research @ Spotify)
 
Search @ Spotify
Search @ Spotify Search @ Spotify
Search @ Spotify
 
An introduction to system-oriented evaluation in Information Retrieval
An introduction to system-oriented evaluation in Information RetrievalAn introduction to system-oriented evaluation in Information Retrieval
An introduction to system-oriented evaluation in Information Retrieval
 
Social Media and AI: Don’t forget the users
Social Media and AI: Don’t forget the usersSocial Media and AI: Don’t forget the users
Social Media and AI: Don’t forget the users
 
Describing Patterns and Disruptions in Large Scale Mobile App Usage Data
Describing Patterns and Disruptions in Large Scale Mobile App Usage DataDescribing Patterns and Disruptions in Large Scale Mobile App Usage Data
Describing Patterns and Disruptions in Large Scale Mobile App Usage Data
 
Evaluating the search experience: from Retrieval Effectiveness to User Engage...
Evaluating the search experience: from Retrieval Effectiveness to User Engage...Evaluating the search experience: from Retrieval Effectiveness to User Engage...
Evaluating the search experience: from Retrieval Effectiveness to User Engage...
 
A Journey into Evaluation: from Retrieval Effectiveness to User Engagement
A Journey into Evaluation: from Retrieval Effectiveness to User EngagementA Journey into Evaluation: from Retrieval Effectiveness to User Engagement
A Journey into Evaluation: from Retrieval Effectiveness to User Engagement
 
From “Selena Gomez” to “Marlon Brando”: Understanding Explorative Entity Search
From “Selena Gomez” to “Marlon Brando”: Understanding Explorative Entity SearchFrom “Selena Gomez” to “Marlon Brando”: Understanding Explorative Entity Search
From “Selena Gomez” to “Marlon Brando”: Understanding Explorative Entity Search
 
How Big Data is Changing User Engagement
How Big Data is Changing User EngagementHow Big Data is Changing User Engagement
How Big Data is Changing User Engagement
 
Measuring user engagement: the do, the do not do, and the we do not know
Measuring user engagement: the do, the do not do, and the we do not knowMeasuring user engagement: the do, the do not do, and the we do not know
Measuring user engagement: the do, the do not do, and the we do not know
 
An Engaging Click ... or how can user engagement measurement inform web searc...
An Engaging Click ... or how can user engagement measurement inform web searc...An Engaging Click ... or how can user engagement measurement inform web searc...
An Engaging Click ... or how can user engagement measurement inform web searc...
 
Social Media News Communities: Gatekeeping, Coverage, and Statement Bias
 Social Media News Communities: Gatekeeping, Coverage, and Statement Bias Social Media News Communities: Gatekeeping, Coverage, and Statement Bias
Social Media News Communities: Gatekeeping, Coverage, and Statement Bias
 
On the Reliability and Intuitiveness of Aggregated Search Metrics
On the Reliability and Intuitiveness of Aggregated Search MetricsOn the Reliability and Intuitiveness of Aggregated Search Metrics
On the Reliability and Intuitiveness of Aggregated Search Metrics
 
Penguins in Sweaters, or Serendipitous Entity Search on User-generated Content
 Penguins in Sweaters, or Serendipitous Entity Search on User-generated Content Penguins in Sweaters, or Serendipitous Entity Search on User-generated Content
Penguins in Sweaters, or Serendipitous Entity Search on User-generated Content
 

Recently uploaded

overview of Virtualization, concept of Virtualization
overview of Virtualization, concept of Virtualizationoverview of Virtualization, concept of Virtualization
overview of Virtualization, concept of VirtualizationRajan yadav
 
Benefits of Fiber Internet vs. Traditional Internet.pptx
Benefits of Fiber Internet vs. Traditional Internet.pptxBenefits of Fiber Internet vs. Traditional Internet.pptx
Benefits of Fiber Internet vs. Traditional Internet.pptxlibertyuae uae
 
SQL Server on Azure VM datasheet.dsadaspptx
SQL Server on Azure VM datasheet.dsadaspptxSQL Server on Azure VM datasheet.dsadaspptx
SQL Server on Azure VM datasheet.dsadaspptxJustineGarcia32
 
Generalities about NFT , as a new technology
Generalities about NFT , as a new technologyGeneralities about NFT , as a new technology
Generalities about NFT , as a new technologysoufianbouktaib1
 
Mary Meeker Internet Trends Report for 2019
Mary Meeker Internet Trends Report for 2019Mary Meeker Internet Trends Report for 2019
Mary Meeker Internet Trends Report for 2019Eric Johnson
 
APNIC Update and RIR Policies for ccTLDs, presented at APTLD 85
APNIC Update and RIR Policies for ccTLDs, presented at APTLD 85APNIC Update and RIR Policies for ccTLDs, presented at APTLD 85
APNIC Update and RIR Policies for ccTLDs, presented at APTLD 85APNIC
 
Google-Next-Madrid-BBVA-Research inv.pdf
Google-Next-Madrid-BBVA-Research inv.pdfGoogle-Next-Madrid-BBVA-Research inv.pdf
Google-Next-Madrid-BBVA-Research inv.pdfMaria Adalfio
 
Tungsten Webinar: v6 & v7 Release Recap, and Beyond
Tungsten Webinar: v6 & v7 Release Recap, and BeyondTungsten Webinar: v6 & v7 Release Recap, and Beyond
Tungsten Webinar: v6 & v7 Release Recap, and BeyondContinuent
 
如何办理朴茨茅斯大学毕业证书学位证书成绩单?
如何办理朴茨茅斯大学毕业证书学位证书成绩单?如何办理朴茨茅斯大学毕业证书学位证书成绩单?
如何办理朴茨茅斯大学毕业证书学位证书成绩单?krc0yvm5
 
Section 3 - Technical Sales Foundations for IBM QRadar for Cloud (QRoC)V1 P10...
Section 3 - Technical Sales Foundations for IBM QRadar for Cloud (QRoC)V1 P10...Section 3 - Technical Sales Foundations for IBM QRadar for Cloud (QRoC)V1 P10...
Section 3 - Technical Sales Foundations for IBM QRadar for Cloud (QRoC)V1 P10...hasimatwork
 

Recently uploaded (10)

overview of Virtualization, concept of Virtualization
overview of Virtualization, concept of Virtualizationoverview of Virtualization, concept of Virtualization
overview of Virtualization, concept of Virtualization
 
Benefits of Fiber Internet vs. Traditional Internet.pptx
Benefits of Fiber Internet vs. Traditional Internet.pptxBenefits of Fiber Internet vs. Traditional Internet.pptx
Benefits of Fiber Internet vs. Traditional Internet.pptx
 
SQL Server on Azure VM datasheet.dsadaspptx
SQL Server on Azure VM datasheet.dsadaspptxSQL Server on Azure VM datasheet.dsadaspptx
SQL Server on Azure VM datasheet.dsadaspptx
 
Generalities about NFT , as a new technology
Generalities about NFT , as a new technologyGeneralities about NFT , as a new technology
Generalities about NFT , as a new technology
 
Mary Meeker Internet Trends Report for 2019
Mary Meeker Internet Trends Report for 2019Mary Meeker Internet Trends Report for 2019
Mary Meeker Internet Trends Report for 2019
 
APNIC Update and RIR Policies for ccTLDs, presented at APTLD 85
APNIC Update and RIR Policies for ccTLDs, presented at APTLD 85APNIC Update and RIR Policies for ccTLDs, presented at APTLD 85
APNIC Update and RIR Policies for ccTLDs, presented at APTLD 85
 
Google-Next-Madrid-BBVA-Research inv.pdf
Google-Next-Madrid-BBVA-Research inv.pdfGoogle-Next-Madrid-BBVA-Research inv.pdf
Google-Next-Madrid-BBVA-Research inv.pdf
 
Tungsten Webinar: v6 & v7 Release Recap, and Beyond
Tungsten Webinar: v6 & v7 Release Recap, and BeyondTungsten Webinar: v6 & v7 Release Recap, and Beyond
Tungsten Webinar: v6 & v7 Release Recap, and Beyond
 
如何办理朴茨茅斯大学毕业证书学位证书成绩单?
如何办理朴茨茅斯大学毕业证书学位证书成绩单?如何办理朴茨茅斯大学毕业证书学位证书成绩单?
如何办理朴茨茅斯大学毕业证书学位证书成绩单?
 
Section 3 - Technical Sales Foundations for IBM QRadar for Cloud (QRoC)V1 P10...
Section 3 - Technical Sales Foundations for IBM QRadar for Cloud (QRoC)V1 P10...Section 3 - Technical Sales Foundations for IBM QRadar for Cloud (QRoC)V1 P10...
Section 3 - Technical Sales Foundations for IBM QRadar for Cloud (QRoC)V1 P10...
 

Advertising Quality Science

  • 2. This talk 4-year effort across research, engineering and product at Yahoo to measure the quality of ads served on Gemini, Yahoo native advertising network Not just measuring but taking actions to improve user experience as well as providing feedbacks to advertisers à no deep learning but large scale predictive analytics Focus of the talk: the post-click experience on native ads à the quality of the landing page… if it is seen
  • 4. Online advertising is big business Values in $billions
  • 5. Advertising is how Yahoo (and many other Internet companies) makes money… and what keeps Yahoo services free for its customers 90% of Yahoo’s revenue is from advertising: 2016 search advertising revenue – $2.67B (52% of total revenue) 2016 display including native advertising – $1.98B (38% of total) Scale: billion ads served daily (Source: Yahoo 2016 10k annual report)
  • 6. Online advertising is about connecting supply & demand Search Native Display Video Brand Direct Response Yahoo own & operated sites Publisher Partners SUPPLY (publishers) DEMAND (advertisers)
  • 7. Advertising (ad) quality science Develop predictive models that characterise the quality of ads shown to and clicked by users. Maximise revenue and guaranteeing ROI to advertisers without negatively impacting user experience. Publishers Advertisers Users Ad inventory ad network Being able to help advertisers improve the quality of their ads
  • 8. Ad Quality: Scope Non-intentional Ad quality Intentional Ad compliance
  • 9. Major shift in how users access the Internet comScore 2015 UK Internet users
  • 11. The quality of the post-click experience the quality of the landing page on mobile
  • 12. The post-click experience: Dwell time dwell time proxy of post-click ad quality experience predictive post-click ad quality models ad quality ratings & recommendations accidental clicks identification proxy of accidental clicks Metric Ad serving User engagement Advertiser ROI
  • 13. dwell time proxy of post-click ad quality experience predictive post-click ad quality models ad quality ratings & recommendations accidental clicks identification proxy of accidental clicks Metric Ad serving User engagement Advertiser ROI The post-click experience journey
  • 14. Quality of the post-click experience Best experience is when conversion happens No conversion does not mean a bad experience Proxy metric of post-click experience: dwell time on the ad landing page tad-click tback-to-publisher dwell time = tback-to-publisher – tad-click Positive post-click experience (“long” clicks) has an effect on users clicking on ads again
  • 15. dwell time proxy of post-click ad quality experience predictive post-click ad quality models ad quality ratings & recommendations accidental clicks identification proxy of accidental clicks Metric Ad serving User engagement Advertiser ROI The post-click experience journey
  • 16. Optimise for high quality ads Estimating P(hq|click) = quality score P(dwell time > t) Build predictive models that predict if an ad is of high quality = predicted dwell time above a given threshold t ➔  high quality = high dwell time revenue = 𝓕 (bid, CTR, quality) P(hq|click) logistic regression, gradient descent boosting, random forest, survival random forest
  • 17. Landing page features ●  window_size ●  view_port ●  media_support content mobile support requested information multimedia mobile optimized out-going connectivity interactivity textual content in-coming connectivity ●  description ●  keywords ●  title meta information ●  num_forms ●  num_input_radio ●  num_input_string ●  ... readability multimedia significant effect of text readability and page structure
  • 18. A/B testing dwell time increased by 20% bounce rate decreased by 7% revenue = bid x CTR x quality (Lalmas etal, 2015; Barbieri, Silvestri & Lalmas, 2016)
  • 19. dwell time proxy of post-click ad quality experience predictive post-click ad quality models ad quality ratings & recommendations accidental clicks identification proxy of accidental clicks Metric Ad serving User engagement Advertiser ROI The post-click experience journey
  • 20. Landing page rating: Low, Average or High landing pages quality score q … L H L and H are customisable: e.g., LOW=[0,25%), AVG=[25%,75%], HIGH=(75%,100%] 2 cut-off points (L, H) that divide distribution of quality scores q into 3 regions: -  LOW: q < L -  AVG: L <= q <= H -  HIGH: q > H (L, H) ad ratingq LOW
  • 21. Improving landing pages Exploiting the features for recommending improvements mobile optimized out-going connectivity interactivity textual content in-coming connectivity meta information readability multimedia ●  num_forms ●  num_input_radio ●  num_input_string ●  ... interactivity ●  mediannum_forms ±ε ●  mediannum_input_radio ±ε ●  mediannum_input_string ±ε ●  ... for each feature compute median and confidence interval for each ad feature compute the distance from the confidence interval given an ad num_input_radio num_forms num_input_string ...
  • 22. There might be too few/much textual content There might be too few/many entities There might be too few/many images Landing Page Content n. of words n. of Wikipedia en;;es n. of images Landing Page Layout height/width resizability (fit to mul;ple screen size) Landing Page Structure n. of drop-down menus n. of checkboxes n. of input strings Landing Page Readability content summarizability The height/width of the landing page might be too small/large The landing page might not be adapted to different screen sizes There might be too many drop-down menus There might be too many checkboxes There might be too much information requested from the users The textual content might be further summarised to make it more readable Examples of recommendations
  • 23. dwell time proxy of post-click ad quality experience predictive post-click ad quality models ad quality ratings & recommendations accidental clicks identification proxy of accidental clicks Metric Ad serving User engagement Advertiser ROI The post-click experience journey
  • 24. peak on app X ●  accidental clicks do not reflect post-click experience ●  not all clicks are equal app X The quality of a click on mobile apps peak on app Y dwell time distribution of apps X and Y for a given ad app Y
  • 25. dwell time proxy of post-click ad quality experience predictive post-click ad quality models ad quality ratings & recommendations accidental clicks identification proxy of accidental clicks Metric Ad serving User engagement Advertiser ROI The post-click experience journey
  • 26. Fitting the data into a mixture model The number of mixture components is determined using the BIC criterion which selects the model that fits best the data while avoiding overfitting Time period 1 Time period 2 (after UI change) Bayesian information criterion (BIC) bouncy clicks accidental clicks
  • 27. Accidental clicks threshold for app X Min 1st Quartile Median Mean 3rd Quartile Max Distribution of the medians as computed on the first component of each ad Applications -  discount accidental clicks using economics models -  train click models discarding accident clicks -  input to UI design ads with all three components
  • 28. dwell time proxy of post-click ad quality experience predictive post-click ad quality models ad quality ratings & recommendations accidental clicks identification proxy of accidental clicks Metric Ad serving User engagement Advertiser ROI Ad quality: The post-click experience journey Acknowledgments: Marc Bron, Ayman Farahat, Andy Haines, Miriam Redi, Gabriele Tolomei, Guy Shaked, Ke (Adam) Zhou, Fabrizio Silvestri, Michele Trevisiol, Ben Shahshahani, Puneet M Sangal and many others