The document presents research on measuring the causal impact of online display advertising, focusing on the challenges of selection bias in observational studies and the use of viewability as a mediating variable. It highlights that traditional analysis tends to overestimate the effectiveness of ads due to targeting specific demographics and behaviors. By utilizing viewability in a natural experiment, the research aims to reduce bias and validate results using negative controls across various advertising campaigns.

1. MLCONF SEATTLE — MAY 1, 2015
A large scale online natural experiment
Measuring causal impact of display ads
Robert Moakler — rmoakler@stern.nyu.edu | robert@integralads.com
@ MLconf Seattle 2015

2. MLCONF SEATTLE — MAY 1, 2015
The $100+ billion question!
Does online advertising really work?
$104.57
$120.05
$140.15
$160.18
$178.45
$196.05
$213.89Digital ad spending!
% change!
2012 2013 2014 2015 2016 2017 2018!
Source: www.emarketer.com, “Global Ad Spending Growth to Double This Year”
20.4%
14.8% 16.7% 14.3%
11.4% 9.9% 9.1%

3. MLCONF SEATTLE — MAY 1, 2015
The $100+ billion question!
Does online advertising really work?

4. MLCONF SEATTLE — MAY 1, 2015
The $100+ billion question!
Does online advertising really work?
Do online ads cause you to take some action?

5. MLCONF SEATTLE — MAY 1, 2015
Measuring causal impact!
Option 1: Randomized A/B test
• Pros
– If setup correctly, gives unbiased
causal estimates
• Cons
– Control ads cost as much as real
ones
– Planned before campaign starts
– Coordination of multiple media
partners
– Too many levers to test them all
Campaign AdPSA

6. MLCONF SEATTLE — MAY 1, 2015
Measuring causal impact!
Option 2: Observational study
• Pros
– Cheap
– Flexible
• Cons
– Enormous amount of selection bias

7. MLCONF SEATTLE — MAY 1, 2015
Confounding in digital advertising campaigns!
• Why is there selection bias in observational techniques?
– Online ads are targeted to specific segments of the population based on
particular demographics, user interests and behaviors, etc.
– Targeting ads to specific populations makes comparing users that have
received an ad to those that did not very problematic; estimates of causal
impact will be overestimated.

8. MLCONF SEATTLE — MAY 1, 2015
Confounding in digital advertising campaigns!
• Why is there selection bias in observational techniques?
W
User
features
A
Served
ads
Y
Convert

9. MLCONF SEATTLE — MAY 1, 2015
Confounding in digital advertising campaigns!
• Why is there selection bias in observational techniques?
W
User
features
A
Served
ads
Y
Convert
Unless we know what information
targeters are using, we will never be
able to fully adjust for selection bias.

10. MLCONF SEATTLE — MAY 1, 2015
Viewability!
• Web page layout, ad placement details, and user browsing behavior
and setup can all impact the way in which ads are seen online.
– Some ads are served far down on the page (below the fold)
– Ads can be loaded in hidden tabs or windows
– Users may not stay on a page long enough for it to finish loading

11. MLCONF SEATTLE — MAY 1, 2015
Viewability!

12. MLCONF SEATTLE — MAY 1, 2015
Viewability!

13. MLCONF SEATTLE — MAY 1, 2015
Viewability!

14. MLCONF SEATTLE — MAY 1, 2015
Viewability as a natural experiment!
Introduce a mediating variable — viewability
W
User
features
A
Served
ads
Y
Convert
V
Viewable
ad

15. MLCONF SEATTLE — MAY 1, 2015
Methodology!
Conversion (Y=1)
Web page visit
Effect window
T0
Untreated user
Treated user
Viewable ad (V=1)
Unviewable ad (V=0)

16. MLCONF SEATTLE — MAY 1, 2015
Methodology!
Conversion (Y=1)
Web page visit
Effect window
T0
Untreated user
Treated user
Viewable ad (V=1)
Unviewable ad (V=0)
Parameter of interest

17. MLCONF SEATTLE — MAY 1, 2015
Data!
• Seven display advertising campaigns run during the 4th quarter of
2014
– Diverse industries such as auto insurance, beauty products, finance, and
online marketing
– 3 million - 29 million impressions
– 2,000 - 2 million conversions

18. MLCONF SEATTLE — MAY 1, 2015
Using viewability as a natural experiment!
Compared to the naïve analysis of comparing users that were served and
not served ads, we find a drastic decrease in estimated lift when utilizing
viewability.

19. MLCONF SEATTLE — MAY 1, 2015
Validation!
• How do we know a reduction in lift means our new estimates are
correct?
• Use negative control tests
– Use the impressions of one campaign to predict an unrelated conversion

20. MLCONF SEATTLE — MAY 1, 2015
Validation!
• How do we know a reduction in lift means our new estimates are
correct?
• Use negative control tests
– Use the impressions of one campaign to predict an unrelated conversion
W
User
features
A
Served
ads
Y
Convert
V
Viewable
ad
Y-
Unrelated
outcome

21. MLCONF SEATTLE — MAY 1, 2015
Validation!
Focusing on Campaign B from the previous example, we measure the
ads’ impact on unrelated outcomes

22. MLCONF SEATTLE — MAY 1, 2015
Bias in the natural experiment!
• We don’t see zero effect on many of our negative controls
– There can be other factors that affect viewability and conversion that we don’t
account for

23. MLCONF SEATTLE — MAY 1, 2015
Bias in the natural experiment!
• We don’t see zero effect on many of our negative controls
– There can be other factors that affect viewability and conversion that we don’t
account for
W
User
features
A
Served
ads
Y
Convert
V
Viewable
ad
W’
User
features

24. MLCONF SEATTLE — MAY 1, 2015
Bias in the natural experiment!
• We don’t see zero effect on many of our negative controls
– There can be other factors that affect viewability and conversion that we don’t
account for
W
User
features
A
Served
ads
Y
Convert
V
Viewable
ad
W’
User
features
Parameter of interest

25. MLCONF SEATTLE — MAY 1, 2015
Validation!
Returning to Campaign B from the previous example, we measure the
ads’ impact on irrelevant outcomes

26. MLCONF SEATTLE — MAY 1, 2015
Summary!
• Viewability enables a natural experiment
– Combines the benefits of A/B tests and observational analysis
– Adjustment for viewability features is easier than adjusting for targeting features
– Results in a large reduction in bias
• Negative controls allow for validation of models when the true value being
estimated is unknown
– As the true effect of a natural experiment is usually unknown, negative controls provide a
method for validation

27. MLCONF SEATTLE — MAY 1, 2015
Versatility!
• Features that can be used in a natural experiment can be found in data sets from a
wide array of industries
– Viewability of stories in a user’s news feed
– Listening to songs on shuffle
– Winning bids in online advertising real-time bidding systems
• Valid negative controls naturally exist in many industries
– Purchasing unrelated products
– Clicking unrelated links

28. MLCONF SEATTLE — MAY 1, 2015
Acknowledgments!
Integral Ad Science
Daniel Hill
Ekaterina Eliseeva
Gijs Joost Brouwer
Kiril Tsemekhman
NYU Stern
Foster Provost
UC Berkley
Alan Hubbard

29. MLCONF SEATTLE — MAY 1, 2015
Thanks!
Robert Moakler — rmoakler@stern.nyu.edu | robert@integralads.com