Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)

Video A/B Testing Deconstructed:
What? How? Where?
(with REAL Data & Examples)
#WistiaFest / @acapland

Hey, I’m Andrew
• Lead the Growth team at Wistia
• Huge marketing & data nerd
• Surfer & snowboarder
• 90’s hip-hop fan

I like testing things

I testing things

We test things, like CTAs

And page layouts

And value props

And self-guided tours

Then we redesign and do it all again

testing works
Because marketers are crazy

WHY aren’t
marketers testing
with video?

Video is more compelling than
static content

SCREEN SHOT OF MY
TALK
That’s my post!
ads

Video is a powerful tool that
humanizes your brand, and compels
your viewers to take action
Marketers don’t understand the
conversion power of video

Action
Increase signups from website
Drive product activation
Accelerate sales process
Decrease churn
Emotional
Storytelling
Humanize brand
Establish connections
Let people get to know you

Action
Increase signups from website
Drive product activation
Accelerate sales process
Decrease churn
You can easily test and
measure these!

You need to be testing
your videos

Two main ways to a/b test with video
Video content Video placement

#WistiaFest / @twitter-handle
1. Why should I test my videos?
2. Where should I test my video?
3. Video testing lessons from Wistia
Let’s dig in more:

1. Why should I test
my videos?

This is the Wistia creative team
Danielle
Dan
Liat
Trevor
Chris

They make awesome things
Danielle
Dan
Liat
Trevor
Chris

Hey we’re redesigning
our About Us page,
should it have a video?
Of course, let’s make
it weird. Everyone
loves weird!
Oh yeah! That
would be sick.
Let’s put it on the
top of the page.
Let’s film it with
our new lens. It’ll
look awesome. Oh, I‘m jacked up!

About Us page

80’s themed pizza
party to celebrate!

JAN FEB MAR APR MAY JUNE JULY
Conversion)Rate
Ruh roh
Conversion Rate

Was it this?

Conversion)Rate
Too late
Test to avoid unexpected results

Conversion)Rate
More please
Test to understand what works

LEARN:
WHAT WORKS
WHAT DOESN’T WORK

Don’t embed it and forget it
See Margot de Cunah’s talk
Don’t publish
videos and just
hope for the best

We get 30k monthly
visits to our page, but
few signups. I wonder
how we could get more?
And what video
might get us the
most signups?
Hmmn. I wonder if
a video would help?
We could film a few
diﬀerent versions?
Oh, I am jacked up!

1. Set clear goals
2. Brainstorm
3. Rank
4. Execute (quickly)
5. Iterate

Sometimes your most boring ideas
are the most eﬀective

You need to be
testing your videos
Learn wicked fast

2. Where should I test
my videos?

Hint: not everywhere

Test in actionable places
where you can learn

Also, somewhere that gets a
decent amount of traﬀic

Product page(s)
Email
About us page
Pricing page
Onboarding flow
Homepage

Optimizely
Sample size calculator - wi.st/sizecalc
A/B confidence calculator - wi.st/statsig
How to run an a/b test

3. Lessons we learned from
testing video at Wistia

Reminder
There are two main ways you can a/b test with video
Video content Video placement

1. Set clear goals
2. Brainstorm
3. Rank
4. Execute (quickly)
5. Iterate
Remember the system

Test 1: Welcome to Wistia page

Test 1: Is this the best video for our goal?

Test 1: Version A

Test 1: Welcome to Wistia page
Hypothesis:
Teaching people about
the tool will increase
account activations.

Test 1: What do you think?
8% increase in
activations

Well, like duh!

Learnings
Straightforward educational
content is more helpful
(for new users)

Test 1: Educational follow-up
Hypothesis:
Highlighting product
diﬀerentiation will
increase activations

Test 1: Follow-up

3% decrease in
activations

Learnings
Teaching how to use tool more
helpful than highlighting features
(for new users).

Test 2: Wistia product page

Test 2: Should this page have a video?

Test 2: Automated Emails

Test 2: Product Page
Hypothesis:
Static content above
the fold may be more
eﬀective.

9% increase in
signups!

Test 2: Learnings
New visitors may want to
browse quickly on the
product page

Test 3: Onboarding flow

Test 3: Is this the best video?

Test 3: Onboarding flow
Hypothesis:
Sharing how to use all
the features will
increase action with
new users.

30% increase in
usage!

Test 3: Learnings
Reinforced our “how to” learning.
Showing better than telling.
Context is king.

You should be testing your videos
Everyone wins!

RECAP
1. Set clear goals
2. Test to learn faster & understand impact
3. Test where it’s actionable (not everywhere)
4. Test your way to success

Use technology to find hidden gems
Collect data (Google Analytics to ID high traﬀic pages)
Watch users (FullStory, Inspectlet)
Survey visitors (Qualaroo, Intercom)

Optimizely
Sample size calculator - wi.st/sizecalc
A/B confidence calculator - wi.st/statsig
A/B Testing Resources

Thank you
Ask me questions anytime:
andrew@wistia.com
@acapland

Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)

More Related Content

What's hot

Viewers also liked

Similar to Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)

Recently uploaded

Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)