Beyond the Button: Tests that actually move the needle - Karen Hopper, Razorfish

KEYNOTE Beyond the Button:
Tests that actually
move the needle
WASHINGTON D.C. ~ JULY 7 - 8, 2022
DIGIMARCONWASHINGTONDC.COM | #DigiMarConWashingtonDC
Karen Hopper
ASSOCIATE DIRECTOR,
PERFORMANCE STRATEGY
RAZORFISH

@nochillfilter
What people think I do

@nochillfilter
What I actually do

@nochillfilter
How do we know if our marketing is
successful?

@nochillfilter
Your ads are awesome!
Congratulations - people are clicking!

@nochillfilter
Your emails are awesome!
Congratulations - people are opening!

@nochillfilter
Markus Spiske on Unsplash

@nochillfilter
But… people do not convert on an ad.

@nochillfilter
Or in an email.

@nochillfilter
Submit a lead form

@nochillfilter
Make a purchase

@nochillfilter
They can only
convert once
they get to the
landing page!
Or… in >88% of
cases, they don’t
convert.
Source: unbounce

@nochillfilter
@nochillfilter
The Case for
Optimization

@nochillfilter
@nochillfilter
Photo by Jp Valery on Unsplash
@nochillfilter

@nochillfilter
We can maximize
our budgets and
efficiency by
improving landing
page conversion
rates
Photo by Isaac Smith on Unsplash @nochillfilter

@nochillfilter
AND don’t forget that landing page
experience is part of the Quality Score:
A good landing page can have a
positive impact on Google Ads Quality
Score, meaning lower CPCs and more
bid wins - which translates to more sales.

@nochillfilter
So how do we make our landing page
better and how do we know it’s doing
better?

@nochillfilter
@nochillfilter
A/B/x testing,
of course!

@nochillfilter
@nochillfilter
Step 1: Start with the data
and your own observations
- What seems to be
working and for whom?
- What does the audience
interact with, or ignore?

@nochillfilter
@nochillfilter
Mobile users have smaller
purchase amounts and a
lower conversion rate
An example!
@nochillfilter

@nochillfilter
@nochillfilter
Step 2: Take your
observations and
brainstorm why
differences exist -
and what’s in your
power to change

@nochillfilter
Mobile users have smaller average
purchase amounts, could we nudge
mobile users to larger purchases by
suggesting different products,
reframing our benefits, or use social
pressure?
An example!

@nochillfilter
@nochillfilter
Step 3: Organize your
ideas by theme and write
a hypothesis for each one.

@nochillfilter
@nochillfilter
How to formulate a hypothesis:
Changing __benefits module___ to __include
mobile app__ will impact mobile
conversions_ because it demonstrates ease
of use for mobile-first consumers.

@nochillfilter
@nochillfilter
A well crafted hypothesis will
always yield learnings, even if
the test fails.
It will also help you identify the
test variables and stay focused.

@nochillfilter
@nochillfilter
Step 4: Prioritize your ideas
based on how much impact
you think they’ll have and
how hard it will be to run the
test.

@nochillfilter
@nochillfilter
Spreadsheets are great for this!

@nochillfilter
@nochillfilter
But how do we
prioritize?

@nochillfilter
@nochillfilter
Assign an effort-benefit score

@nochillfilter
@nochillfilter
How to
calculate
effort-benefit
score
Expected Impact +
Learning Priority -
Technical Effort
Total score

@nochillfilter
@nochillfilter
How to
calculate
effort-benefit
score
Expected Impact (3) +
Learning Priority (2) -
Technical Effort (2)
Total score (3)

@nochillfilter
@nochillfilter
Now you can sort by total score or expected impact!

@nochillfilter
@nochillfilter
Step 5: Test and repeat!

@nochillfilter
Develop and
prioritize test ideas
Plan and execute a
specific test based on
a hypothesis
Evaluate the test and
learnings to influence
future test ideas

@nochillfilter
The Science Art of Optimization

@nochillfilter
@nochillfilter
I see this a lot…
Testing is risky so...

@nochillfilter
Photo by Jud Mackrill on Unsplash
Photo by Mimi Thian on Unsplash

@nochillfilter
Guess what? Nobody cares!

@nochillfilter
Guess what? Nobody cares!*

@nochillfilter
Photo by Melani Sosa on Unsplash
@nochillfilter

@nochillfilter
Yes, we COULD
change everything,
but we SHOULDN’T
change everything
- otherwise, how
will we know what’s
working, so we can
do more of it?

@nochillfilter
“best practices”

@nochillfilter
“test practices”

@nochillfilter
What is working to convert users in
other channels?

@nochillfilter
14%
of businesses say they share experimentation results
across departments.
Only
Source: Speero Agency

@nochillfilter
Sees a display ad….,
googles a specific
product….., clicks through
a paid result….., browses
the site, reads an
article….., signs up for
email…., clicks on an
email…… makes a
purchase.
Mark König on Unsplash

@nochillfilter
What are competitors or other industries
doing in their ads and on their landing
pages?

@nochillfilter
Remember this
landing page?
Well… it’s not
mine…

@nochillfilter
Our audience doesn’t live in a
bubble, why should our ideas?

@nochillfilter
Get familiar with Industry
Research + Psychology:
Why we buy things, how we
make decisions, UX design, etc. -
Marketing 101

@nochillfilter
@nochillfilter
A Winning
Approach

@nochillfilter
“But how many variants can
we test at once, and how long
will it take to run test?”

@nochillfilter
Even a massive difference in performance doesn’t
mean much if our audience isn’t big enough.
Going from 3 conversions to 6 is a 100% increase, but
it probably isn’t giving you a statistically significant
result.

@nochillfilter
@nochillfilter
Needed Sample Size (per variant):
https://www.optimizely.com/sample-size-calculator/
Try to let your tests run for at least 2 weeks
to normalize any weirdness in your data!

@nochillfilter
@nochillfilter
What if I need a bigger sample size?

@nochillfilter
Can you…
● Run the test in several campaigns
instead of just in one?
● Add additional segments or
channels to your experiment?
● Remove one or more variants?
● Or…. Can you test something
where you think you’ll see a
bigger impact?
@nochillfilter

@nochillfilter
It is the most tempting...
Every testing platform
has a dashboard that
offers a glance into how
your test is doing right
now - which makes it so
hard to not snoop at the
results and check it all
the time.

@nochillfilter
But every time you check it...
You’re performing another
statistical test on the results.
This is called “repeat significance
testing” and can cause us to find
the one moment in time that results
just happen to be significant.
If we had stopped our experiment
after seeing the “significant” results
after a few days, we would have
missed that the test variant was
actually the loser.

@nochillfilter
https://www.mrss.com/toolshed/chi-squared-test/

@nochillfilter
@nochillfilter
This means that if we repeat the
experiment over and over again to the
same group of people under the same
conditions, 19 out of 20 times, we can
expect the same outcome will occur.
There is some risk that random chance will
favor one version over another, but by
using the 95% confidence interval, we are
reducing our risk that we are picking the
wrong condition.
Statistical Significance:
95% Confidence

@nochillfilter
NEVER SAY A VARIANT WON IF THERE
WAS NO STATISTICAL SIGNIFICANCE.
@nochillfilter

@nochillfilter
@nochillfilter
Building a Data
Driven Optimization
Program

@nochillfilter
DATA DRIVEN
OPTIMIZATION
AUDIENCE SIZE
supports the tests
you want to run
TAILOR the tests to
your program; make
sure it’s a good fit
ADJUST based
on learnings
BENEFIT: Can this test
reasonably have a big
impact and can we
measure it?
COST: How long/$$ will
it take to build and
design?
EMPHASIZE
program goals
(KPIs)

@nochillfilter
SQUIRREL!
@nochillfilter

@nochillfilter
Remember that you shouldn’t run multiple tests
on the same people at the same time, otherwise
you won’t know which test is having the impact
you’re seeing.
Headline
Test Default
Payment
Options
CTA copy
Hero
Image
June July August September

@nochillfilter
Every test should
have:
+ A unique name
+ A judgement on how
difficult it would be to
implement the changes
+ The number of users or
clicks you’ll test on
before evaluating
+ The main metric(s) for
success - usually CTR
and Conversion Rate.
@nochillfilter

@nochillfilter
@nochillfilter
And remember...
Once you’re ready to launch your test,
let go of your assumptions about what
will work well and let the test results
speak for themselves (via statistical
significance!).

@nochillfilter
#HEROCONF
Discovery Hypotheses and Testing
Make observations (with data!) based on what
you know about your marketing and your
audiences and line them up with your goals.
Research
Learn more about why people
behave in general and use that to
inform your hypotheses, and to
generate ideas for ways to
improve performance.
Prioritize and Plan
Document and schedule your ideas,
and begin testing.
4.) Evaluate and Repeat
Validate your hypotheses, build on
what works, and go back to the
observation and ideation stage as
necessary.
Generate Ideas

@nochillfilter
“Never stop testing, and your
advertising will never stop
improving.”
- David Ogilvy

@nochillfilter
@nochillfilter
@nochillfilter @lalaforte

Beyond the Button: Tests that actually move the needle - Karen Hopper, Razorfish

Recommended

Recommended

More Related Content

Similar to Beyond the Button: Tests that actually move the needle - Karen Hopper, Razorfish

Similar to Beyond the Button: Tests that actually move the needle - Karen Hopper, Razorfish (20)

More from DigiMarCon - Digital Marketing, Media and Advertising Conferences & Exhibitions

More from DigiMarCon - Digital Marketing, Media and Advertising Conferences & Exhibitions (20)

Recently uploaded

Recently uploaded (20)

Beyond the Button: Tests that actually move the needle - Karen Hopper, Razorfish