The document discusses a four-year effort at Yahoo focused on enhancing the quality of ads within its native advertising network, Gemini, using large-scale predictive analytics instead of deep learning. It emphasizes the importance of the post-click experience, particularly the quality of landing pages, dwell time, and user engagement which are critical for maximizing revenue and ensuring a positive return on investment for advertisers. The work includes developing predictive models for assessing ad quality, recommending improvements for landing pages, and identifying accidental clicks to better understand user interactions with ads.

1. Advertising Quality Science
Mounia Lalmas

2. This talk
4-year effort across research, engineering and product at Yahoo to
measure the quality of ads served on Gemini, Yahoo native advertising
network
Not just measuring but taking actions to improve user experience as well
as providing feedbacks to advertisers
à no deep learning but large scale predictive analytics
Focus of the talk: the post-click experience on native ads
à the quality of the landing page… if it is seen

3. 3
The advertising world is fun J

4. Online advertising is big business
Values in $billions

5. Advertising is how Yahoo (and many other Internet companies) makes
money… and what keeps Yahoo services free for its customers
90% of Yahoo’s revenue is from advertising:
2016 search advertising revenue – $2.67B (52% of total revenue)
2016 display including native advertising – $1.98B (38% of total)
Scale: billion ads served daily
(Source: Yahoo 2016 10k annual report)

6. Online advertising is about connecting supply & demand
Search
Native
Display
Video
Brand
Direct
Response
Yahoo own &
operated sites
Publisher Partners
SUPPLY
(publishers)
DEMAND
(advertisers)

7. Advertising (ad) quality science
Develop predictive models that characterise the quality of ads shown to and clicked by users.
Maximise revenue and guaranteeing ROI to advertisers without
negatively impacting user experience.
Publishers
Advertisers
Users
Ad
inventory
ad network
Being able to help
advertisers improve the
quality of their ads

8. Ad Quality: Scope
Non-intentional
Ad quality
Intentional
Ad compliance

9. Major shift in how users access the Internet
comScore 2015 UK Internet users

10. Native advertising
(Source: Sharethrough.com & IPG Media Lab Study: Na;ve Adver;sement Effec;veness)
Visually Engaging
Higher user attention
Higher brand lift
Social sharing

11. The quality of the post-click experience
the quality of the landing page on mobile

12. The post-click experience: Dwell time
dwell time
proxy of post-click ad
quality experience
predictive post-click ad
quality models
ad quality ratings &
recommendations
accidental clicks
identification
proxy of accidental
clicks
Metric
Ad serving
User
engagement
Advertiser
ROI

13. dwell time
proxy of post-click ad
quality experience
predictive post-click ad
quality models
ad quality ratings &
recommendations
accidental clicks
identification
proxy of accidental
clicks
Metric
Ad serving
User
engagement
Advertiser
ROI
The post-click experience journey

14. Quality of the post-click experience
Best experience is when conversion happens
No conversion does not mean a bad experience
Proxy metric of post-click experience: dwell time on the ad landing page
tad-click tback-to-publisher
dwell time = tback-to-publisher – tad-click
Positive post-click experience (“long” clicks)
has an effect on users clicking on ads again

15. dwell time
proxy of post-click ad
quality experience
predictive post-click ad
quality models
ad quality ratings &
recommendations
accidental clicks
identification
proxy of accidental
clicks
Metric
Ad serving
User
engagement
Advertiser
ROI
The post-click experience journey

16. Optimise for high quality ads
Estimating P(hq|click) = quality score
P(dwell time > t)
Build predictive models that predict if an ad is of high quality
= predicted dwell time above a given threshold t
➔ high quality = high dwell time
revenue = 𝓕 (bid, CTR, quality)
P(hq|click)
logistic regression, gradient descent boosting, random forest, survival random forest

17. Landing page features
● window_size
● view_port
● media_support
content
mobile
support
requested
information
multimedia
mobile
optimized
out-going
connectivity
interactivity
textual
content
in-coming
connectivity
● description
● keywords
● title
meta
information
● num_forms
● num_input_radio
● num_input_string
● ...
readability multimedia
significant effect of text readability and page structure

18. A/B testing
dwell time increased by 20%
bounce rate decreased by 7%
revenue = bid x CTR x quality
(Lalmas etal, 2015; Barbieri, Silvestri & Lalmas, 2016)

19. dwell time
proxy of post-click ad
quality experience
predictive post-click ad
quality models
ad quality ratings &
recommendations
accidental clicks
identification
proxy of accidental
clicks
Metric
Ad serving
User
engagement
Advertiser
ROI
The post-click experience journey

20. Landing page rating: Low, Average or High
landing
pages quality score q
…
L H
L and H are customisable:
e.g., LOW=[0,25%), AVG=[25%,75%], HIGH=(75%,100%]
2 cut-off points (L, H) that
divide distribution of quality
scores q into 3 regions:
- LOW: q < L
- AVG: L <= q <= H
- HIGH: q > H
(L, H)
ad ratingq LOW

21. Improving landing pages
Exploiting the features for recommending improvements
mobile
optimized
out-going
connectivity
interactivity textual
content
in-coming
connectivity
meta
information
readability
multimedia
● num_forms
● num_input_radio
● num_input_string
● ...
interactivity
● mediannum_forms ±ε
● mediannum_input_radio ±ε
● mediannum_input_string ±ε
● ...
for each feature
compute median and
confidence interval
for each ad feature
compute the distance from
the confidence interval
given an ad
num_input_radio
num_forms
num_input_string
...

22. There might be too few/much textual content
There might be too few/many entities
There might be too few/many images
Landing Page Content
n. of words
n. of Wikipedia en;;es
n. of images
Landing Page Layout
height/width
resizability (fit to mul;ple screen size)
Landing Page Structure
n. of drop-down menus
n. of checkboxes
n. of input strings
Landing Page Readability
content summarizability
The height/width of the landing page might be too small/large
The landing page might not be adapted to different screen
sizes
There might be too many drop-down menus
There might be too many checkboxes
There might be too much information requested from the
users
The textual content might be further summarised to make it more
readable
Examples of recommendations

23. dwell time
proxy of post-click ad
quality experience
predictive post-click ad
quality models
ad quality ratings &
recommendations
accidental clicks
identification
proxy of accidental
clicks
Metric
Ad serving
User
engagement
Advertiser
ROI
The post-click experience journey

24. peak on app X
● accidental clicks do not
reflect post-click
experience
● not all clicks are equal
app X
The quality of a click on mobile apps
peak on app Y
dwell time distribution of apps X and Y for a
given ad
app Y

25. dwell time
proxy of post-click ad
quality experience
predictive post-click ad
quality models
ad quality ratings &
recommendations
accidental clicks
identification
proxy of accidental
clicks
Metric
Ad serving
User
engagement
Advertiser
ROI
The post-click experience journey

26. Fitting the data into a mixture model
The number of mixture components is determined using the BIC criterion
which selects the model that fits best the data while avoiding overfitting
Time period 1
Time period 2 (after UI change)
Bayesian information criterion (BIC)
bouncy clicks
accidental clicks

27. Accidental clicks threshold for app X
Min
1st Quartile
Median
Mean
3rd Quartile
Max
Distribution of the medians as computed on the first
component of each ad
Applications
- discount accidental clicks
using economics models
- train click models
discarding accident clicks
- input to UI design
ads with all three components

28. dwell time
proxy of post-click ad
quality experience
predictive post-click ad
quality models
ad quality ratings &
recommendations
accidental clicks
identification
proxy of accidental
clicks
Metric
Ad serving
User
engagement
Advertiser
ROI
Ad quality: The post-click experience journey
Acknowledgments: Marc Bron, Ayman Farahat, Andy Haines, Miriam Redi, Gabriele Tolomei, Guy Shaked,
Ke (Adam) Zhou, Fabrizio Silvestri, Michele Trevisiol, Ben Shahshahani, Puneet M Sangal and many others