Evolution,
not Revolution.
Why Optimizing
Beats Redesigning.
[presented by] Kate O’Neill
of [meta]marketer
Image credit: funfonix.com

Who am I to make
these claims?
And who are you
to listen?

Why I’m a Believer
• Helped lead rollout through data awareness of
interactive redesign at Netflix that is still standard
for e-commerce
• Oversaw many multivariate tests at Magazines.com
that resulted in 40% lift in conversion rate YOY
• Now own a company that focuses on this kind of
optimization for clients with great results

Why You May
Choose to Believe
• Working in a company torn by politics or
conflicting interests in web design or feature?
• Designing sites freelance and always encountering
the same resistance?
• Running personal web sites and blogs you’d like to
see perform better?

How are most
design decisions made?

HPPO
• Highest
Paid
Person’s
Opinion
• Not irrelevant, but not comprehensive.

By Committee
• “I need my department to be featured on
the home page!”
• “Can we use something other than red for
the Buy buttons?”

Designer’s Aesthetic
• Flash!
• Prettiness over performance

Finger in the Wind
• What seems to be trendy
• What someone mentioned at a party
• What’s reported in the news

In other words...
in a vacuum.

How should design
decisions be made?

You need user input.
You are not your audience.
Even if you are.

You need data.
Users lie.
Aggregate data doesn’t.

Data trumps
opinions.
Even highly paid
ones.
HPPO
Use conversion-related metrics
to determine executive-relevant
strengths and weaknesses of the site.

If it isn’t
interesting
to the user, ditch
it.
Committee
Use engagement metrics
to determine what keeps users on the site.

Think users like
your design?
Prove it.
Design for
Design’s Sake
If your success can’t be measured,
you can’t defend it.
And your input won’t be appreciated.

“Great idea! I’ll
add it to the
testing roadmap.”
Finger in
the Wind
Trendy ideas are worth knowing about.
But they may not work in your environment.

Balance objective
and subjective input
what people think or feel
versus
what people do

Balance qualitative
and quantitative input
what you can intuit
versus
what you can measure

How to Balance
Input Gathering
analytics data
qualitative - - - - - - quantitative
A/B or MVT results
surveys
focus groups
usability
studies / interviews
subjective - - - - - - - - - - - - - - - - - - - - - - - - objective

What to do?
‣ Balance subjective & objective testing
(And know that you may get it wrong)
‣ Find the story behind the story
(But know that you may get it wrong)
‣ Look for a narrative in onsite testing
(But know that you may get it wrong)
‣ Look for the unobvious AND the obvious
(And know that you may get it wrong)

If you’re still going to
get it wrong, why test?
Because you can not only measure lift
when you’re right...
(Woo hoo!)
you can also contain risk when you’re
wrong.
(And it just might save your job.)

Test your way
to greatness.

How a Redesign Can
Go Wrong:
HPPO Edition

Netflix Branding Gaffe
circa 2000
TV?! Static?!
Totally wrong
customer experience
emphasis.

Improved
Playing off of
the movie theater
experience.

How a Redesign Can
Go Wrong:
Usability Edition

Look Inside: Setup

Look Inside: Results
• Visitors more likely to click on Preview offer (4.89% average) vs. Explore
(3.85%)
• Both variations resulted in a drop in conversion
• Drop in conversion only slightly greater with Preview (-.87%) than with
Explore (-.76%)
• Overall, more established titles (Time, Sports Illustrated, People) had
lowest Click % as well as below average drop in conversion of all viewer-
enabled titles
• Somewhat lesser-known titles (SI Kids, Health, StyleWatch, Money) had
highest Click % and higher than average drop in conversion of all viewer-
enabled titles

What to do?
(Once more with feeling!)
‣ Balance subjective & objective testing
(And know that you may get it wrong)
‣ Find the story behind the story
(But know that you may get it wrong)
‣ Look for a narrative in onsite testing
(But know that you may get it wrong)
‣ Look for the unobvious AND the obvious
(And know that you may get it wrong)

If all else fails:
[meta]marketer can help.
Thank you!
Kate O’Neill, Founder / CEO
kate@metamarketer.com
615-852-META
twitter: @kateo / @metamarketer