Take the guesswork out of driving results from your Streaming campaigns, and discover how the always-on approach can deliver the metrics needed for a full picture across your entire campaign.
In this webinar, our experts at Tinuiti and our Streaming agency, Bliss Point Media, with guest global market research company, Forrester, explore Streaming advertising marketing trends and the importance of incrementality testing in driving sales.
2. Today’s Logistics
Kerry Mallett
Senior Content Specialist,
Webinar
Recording & slides will be in your inbox
shortly after the session
Download relevant resources
Join our chat and submit questions for live Q&A
Log back anytime with same link
Join our poll questions - including if you want to talk to
our teams after!
3. WE’VE MASTERED THE
MEDIA THAT
MATTERS MOST
Tinuiti is the largest independent performance
marketing firm across Streaming TV and the
Triopoly of Google, Facebook, Amazon.
Employees
Media managed
Award-winning
culture*
$3B
#1
1000+
5. Our Speakers
PHILIP RUNDALL
Senior Manager of
Economics
TINA MOFFETT
Principal Analyst
LAUREN WOLFEN
VP of Client Strategy &
Analytics
WITH GUEST SPEAKER
6. Agenda
● Improve Your Streaming With Incrementality Testing
● Introduction To Randomized Controlled Experiments
● Incrementality Case Study
● Traditional And Updated Incrementality Testing
● What Is Incrementality & Always-On Incrementality?
● What This Means For Modern Marketers
6
7. What is your comfort level with
experimental methods from other
channels?
● Very comfortable
● Somewhat comfortable
● Not comfortable
POLL
8. How much of a priority is building
experiments to measure incrementality?
● A big priority
● Somewhat a priority
● Not a priority
POLL
9. Improve Your Streaming With
Incrementality Testing
Tina Moffett
Principal Analyst
B2C Marketing Executives
23. Experimental Methods in OTT Can Provide More Efficient
Decisions
23
● As omnichannel marketers, we
need to understand
incrementality throughout the
funnel to drive optimal
investment decisions.
● OTT can serve as a full-funnel
marketing tool, broadening
reach and filling the top of the
funnel while producing efficient
acquisition in its own right.
Awareness
Acquisition
24. There Are A Few Techniques To Understand
Incrementality Throughout The Funnel
Experimental
Methods
Quasi-Experimental
Methods
Observational
Methods
25. Experimental Methods Are The Gold Standard
25
OTT Incrementality Tests benefit from being experiments.
● Test designed to meet requirements.
● Proper randomization is done.
● Accurately measures incrementality on an OTT publisher.
The basic principle of the test
involves creating two distinct
impression groups: a control group,
and an exposed group.
TV lift tests can often be run with a
conservative budget and a relatively
short time frame to produce
statistically significant results that
equate to a measured lift in response.
Measurement
To understand the incrementality of
TV, we measure and compare the
responses of the control group vs. the
group that saw the commercials over
a distinct time frame and for a
particular KPI.
Test Structure Implementation
26. Test Overview
Client industry: Wedding
services
KPI: Signups
Test duration: 3 weeks
Property: Hulu Reality Cluster
Total test spend: $30k
Spend ratio: 80% / 20%
(exposed / control)
Findings
Based on the difference in
response rates, the exposed
group was ~6x more likely to
register for an account than
the control group.
Control Group Exposed Group
N= 159k
N= 102 signups
N= 36k
N= 4 signups
4.0
3.8 x 10-5
t-Stat
p-Value
CASE STUDY
Streaming Audience Holdout
Audience Sample
Sample Response
Response Rate
1.1 x
10-4
6.4 x 10-4
27. Experiments Have Their Practical Limitations
27
Truth in
the test
Truth in
real life
Generalization
Internal
Validity
External
Validity
● Money spent on placebo ads.
● Limited OTT partner capabilities.
● Incrementality changes over time.
● Results to not apply to other publishers.
Incrementality Test Drawbacks Problems With External Validity
28. Quasi-Experimental Methods Solve Drawbacks And
Maintain Validity
28
● Quasi-experimental methods are used
when randomization is not possible.
● Results are still valid even if
randomization is not used.
● Commonly used in the social sciences.
29. Always-On Incrementality Allows For Costless,
Continuous, and Campaign-Wide Insights
29
Incrementality: Measuring a funnel
step or action that would not have
occurred without a specific
interaction, such as seeing an
advertisement.
Always-On Incrementality: Constantly
being able to measure the
incrementality of a marketing
campaign. Not just a static
measurement based on a test that
has been run in the past.
Incremental
Conversions
Converted because
they saw the OTT
impression
Background
Responses
Would have
converted
regardless of
exposure to
impression
Video Impressions That Converted
30. How Always-On
Incrementality Works
30
● Impressions are served on
the same publishers by
multiple clients.
● Behaviorally match
impressions to create
control groups.
● Quantify incrementality.
Daily Measurement Process
31. What Does This Mean For A Marketer?
31
● With always-on
incrementality, we can
take the guesswork out of
driving a campaign as a
marketer.
● No longer need to apply a
haircut to the results to
get the “real” metrics, now
know the full picture
based on always-on
incrementality.
32. Key Takeaways Tinuiti/Bliss Point Media:
1. Quasi-experimental methods in OTT can lead
more efficient decisions.
2. Always-on incrementality allows for costless,
continuous, and campaign-wide insights.
Forrester:
1. Create a testing culture in your organization by
sharing success stories and the economic value
of marketing experiments.
2. Build a baseline of analytics literacy. Start with the
constructs of what makes a good test: statistical
significance, population representation, and clear
testing parameters.