Are my ads causing incremental actions to be taken? This is the key question we’d like our media measurement to address. However, a lion’s share of solutions are still based off correlation and cannot prove that your ad caused the action or if you took credit for an action that would have occurred regardless of advertising. Lift testing is the solution to help you escape the biases of correlated based methods, allowing you to fully understand if your ads are causing incremental actions to be taken. Learn more about lift testing, common pitfalls to avoid and how you can get started.

1. Teresa Lewis
SR. CONSULTANT, ADVANCED MEASUREMENT SERVICES,
ADOBE
CHICAGO, IL ~ JUNE 20 - 21, 2019 | DIGIMARCONMIDWEST.COM
#DigiMarConMidwest
Do You Even Lift?:
Incrementality & Lift Testing
Explained
KEYNOTE

2. © 2019 Adobe. All Rights Reserved. Adobe Confidential.
Why Lift?
Discuss best practices
Learn how to get started
2
A G E N D A

3. © 2019 Adobe. All Rights Reserved. Adobe Confidential.
WHY DO WE ADVERTISE?
3

4. © 2019 Adobe. All Rights Reserved. Adobe Confidential.
Digital Attribution is Often Based on Correlation
4
+ =
$$$
$$$

5. © 2019 Adobe. All Rights Reserved. Adobe Confidential.
Digital Attribution is Often Based on Correlation
5
+ =
$$$
$$$

6. © 2019 Adobe. All Rights Reserved. Adobe Confidential.
Correlation Does Not Equal Causation
6
+ =
$$$
$$$

7. © 2019 Adobe. All Rights Reserved. Adobe Confidential.
Audience Bias
LIKELIHOODTO
BUY
Loyalist Shopping
Around
Competitor’s
Loyalist
Beware of Correlated Measurement Biases
Media Cost
IMPRESSION
VOLUME
1x
Rate
2x
Rate
50%
Conversion Rate
Low Rate Bias

8. © 2019 Adobe. All Rights Reserved. Adobe Confidential.
Correlation vs. Causation
8
Tactic Ad Spend
Buyers After
Ad Exposure
CPA
TACTIC A
TACTIC B
$100
$100
$12.50
$20
Incremental
Buyers
Cost Per
Incremental
$100
$25
Performance Optimization Fallacy in Action

9. © 2019 Adobe. All Rights Reserved. Adobe Confidential.
The Bigger Investment Picture
9
$283
$333
$385
$436
$479
$518
45.9%
50.1%
53.6%
56.6%
58.8%
60.5%
$0
$100
$200
$300
$400
$500
20%
30%
40%
50%
60%
70%
2018 2019 2020 2021 2022 2023
Global Digital Ad Spending (In Billions)
% of Total Media Spend

10. © 2019 Adobe. All Rights Reserved. Adobe Confidential. 10
START LIFTING!

11. © 2019 Adobe. All Rights Reserved. Adobe Confidential. 11

12. © 2019 Adobe. All Rights Reserved. Adobe Confidential.
Control Groups Provide a North Star
12
Natural Converters Actual Lift Driven by Ad
% Ad Conversions - % Control
Conversions
=
Actual Ad Lift

13. © 2019 Adobe. All Rights Reserved. Adobe Confidential.
Define the Goal of Your Lift Test
Set your objective and only test 1 dependent & independent
variable at a time.
Dependent Variable
Will delivering ads
to my lapsed
customers cause
them to purchase?
Learning Objective Independent Variable

14. © 2019 Adobe. All Rights Reserved. Adobe Confidential.
Define the Goal of Your Lift Test
%
L
I
F
T
E
X
P
O
S
E
D
C
O
N
T
R
O
L
+ =
+ =

15. © 2019 Adobe. All Rights Reserved. Adobe Confidential.
Do Lift Test Sizing Before You Start
Assess Feasibility & Estimate Samples Needed for a Statistically
Successful Test
Establish
Behavior
Baseline
Estimate Sample
Sizes
Benchmark

16. © 2019 Adobe. All Rights Reserved. Adobe Confidential.
Sample Size Considerations
16
0
20,000
40,000
60,000
80,000
100,000
120,000
140,000
160,000
180,000
0 0.2 0.4 0.6 0.8 1
As conversion rate goes up,
sample size needed goes
down
Conversion Rate
SampleSize(Uniques)
0
20,000
40,000
60,000
80,000
100,000
120,000
140,000
160,000
180,000
1 1.05 1.1 1.15 1.2 1.25
As lift goes up, sample
size needed goes down
Lift
SampleSize(Uniques)
As exposed ratio goes
up, sample size needed
goes up
0
1,000
2,000
3,000
4,000
5,000
6,000
3 5 7 9 11
Exposed Ratio
SampleSize(Uniques)

17. © 2019 Adobe. All Rights Reserved. Adobe Confidential.
Ensure Your Samples are Randomized
In-Market
Audience
In-Market
Audience
In-Market- Test Brand’s
Loyalists
In-Market- Competitor’s
Loyalists

18. © 2019 Adobe. All Rights Reserved. Adobe Confidential.
Ensure Your Samples are Randomized
In-Market- Test Brand’s Loyalists
In-Market- Competitor’s
Loyalists
In-Market
Audience
In-Market
Audience
Contr
ol
Contr
ol

19. © 2019 Adobe. All Rights Reserved. Adobe Confidential.
Ensure Your Samples are Randomized
In-Market
Audience
In-Market
Audience
In-Market- Test Brand’s Loyalists
In-Market- Competitor’s
Loyalists
Expos
ed
Expos
ed

20. © 2019 Adobe. All Rights Reserved. Adobe Confidential.
Ensure Your Samples are Randomized
In-Market
Audience
Expos
ed
Control Expos
ed
Control
In-Market- Test Brand’s Loyalists
In-Market- Competitor’s
Loyalists

21. © 2019 Adobe. All Rights Reserved. Adobe Confidential.
Verify Efficacy of Randomization
Experiment Designer
Randomization
Expos
ed
Control
Control Group Efficacy Test
Non-Test
Brand Event
Control
Conversion
Rate
Exposed
Conversion
Rate
% Difference
Purchase Rate
at
Retailer
13% 13% No Difference
Purchase Rate
at Home Supply
Store
.17% .17% No Difference
Purchase Rate
at QSR
1% 1% No Difference
% Women in
Group
49% 49% No Difference

22. © 2019 Adobe. All Rights Reserved. Adobe Confidential.
Remember You Are Not a Scientist- This is
Advertising
Isolate your independent variable, while still running an ad
campaign.

23. © 2019 Adobe. All Rights Reserved. Adobe Confidential.
Action & Build off Your Results
Incorporate Learnings Into Your Buying Best
Practices
23

24. © 2019 Adobe. All Rights Reserved. Adobe Confidential. 24
Redbox AdPlacebo Ad
Setup of Redbox Lift Test

25. © 2019 Adobe. All Rights Reserved. Adobe Confidential. 25
1 Create control groups directly in
the Advertising Cloud buying
platform
2 Randomizes audiences at the ad
call for immediately clean
sample groups
3 Real-time results with control to
adjust and optimize strategies
for constant improvement
Experiment Designer
Methodology

26. © 2019 Adobe. All Rights Reserved. Adobe Confidential. 26
The Incrementality Results
Cart Abandoners 1 Cart Abandoners 2
Movie Rental ROAS
Placebo Exposed
LIFT 33%
LIFT 80%

27. © 2019 Adobe. All Rights Reserved. Adobe Confidential. 27
1 Redbox ran media against only
the highest ROAS audience to
capitalize on media
effectiveness
2 Additional audiences to be
tested throughout 2019 using the
same methodology
3 Redbox has a clear test and
learn framework moving forward
to vet new campaign tactics

28. © 2019 Adobe. All Rights Reserved. Adobe Confidential.
Correlation doesn’t equal causation- beware of biases
Gain internal alignment & start lifting to measure
causality
Incorporate best practices to avoid pitfalls
Action off your results
28
K E Y
T A K E A W A Y S