Test ideation is at the core of growing your company’s online sales, especially as consumers’ expectations for ease of use and personalized experiences continue to climb. This session will help you uncover tests to increase key metrics, like conversion rates and average order value, and make shopping on your site a delightful experience for every customer.
2. • Optimizely's top solutions partner for enterprise brands & retailers
• Partner to over 30 clients including Adidas, Tory Burch, BluDot,
Whole Foods, Cost Plus World Market, Express, Patagonia &
many more
• Proud winner of 2015 Opticon Partner of the Year & Most
Dramatic Business Impact
hi. i’m matty.
founder & ceo at
SPRING INTO ACTION
Matty Wishnow
& Leo Wishnow
• 15+ years testing & validating as:
• SVP, Direct to Consumer at
Warner Music Group
• Founder & CEO of Insound.com
• Loves “Less and More” design
principles & advanced baseball stats
• Hates dabbling
3. so, you want
test ideas.
hypotheses learnings investmentsproblemsgoals
wouldn’t you rather a repeatable framework for
producing & prioritizing great test ideas?
SPRING INTO ACTION
4. the ideas will come.
but, let’s start with the framework.
goals problems hypotheses learning
UX
product
merchandising
marketing
prioritized by data validated by data
investments
1
2
3
Ps
pS
ps
PS
2
3
4
1
2
3
4
1
SPRING INTO ACTION
5. unifying.
wait like this applies to everything?
#1 Everything is an experiment
Banner/hero changes for new
seasons/campaigns.
Merchandising changes.
Copy changes.
Redesigns & replatforms.
New SaaS additions.
Everything.
#2 Questions to live by:
What problems are the
changes solving?
How will you know if the
changewas successful?
SPRING INTO ACTION
6. how to
communicate goals
good example:
In 2016, we want to increase
smart phone purchase
conversion rates by 10% y/o/y.
bad example:
We want to improve the mobile
user experience.
Ask yourself:
• Are they widely understood?
• Are they actionable?
• Are they measurable?
• Do they have refined baselines/targets?
• Are they time-based?
SPRING INTO ACTION
7. how to
capture problems
good example:
Our customers have a problem
understanding the difference
between and value proposition
of products on PLPs.
bad example:
I think we need a responsive
website.
Ask yourself:
• Why is it a problem?
• How does the problem impact the
business or end user?
• Is it a root problem?
• Do you have data to confirm that it is
a significant problem?
SPRING INTO ACTION
8. how to
articulate a hypothesis
good example:
I believe that if we add geo-targeted
shipping commitments on the shipping
page of checkout, customers will have
more confidence in ordering from us.
If I am right we will see increased conversion
& reduced abandonment at the shipping page
of checkout.
bad example:
I think we need to redesign
checkout.
Try to:
• Be specific about the change you are
contemplating.
• Describe what you expect the result of
that change to be.
• Clearly define the metric you can most
confidently measure success by.
• Start with:
• I believe that…
• If I am right then…
SPRING INTO ACTION
9. Be consistent about requiring
all changes get mapped to
problems & hypotheses.
2
what do you do once
it’s all mapped?
Determine if additional
problem research is needed.
1
hypotheses learnings investmentsproblemsgoals
Determine what should be
tested vs “just do it”.
3
Review & iterate regulary.4
SPRING INTO ACTION
10. I believe that if we remove the “New
Customers” section on the interstitial
page between the ShoppingBag and
checkout, we will reduce friction to
continue to checkout for non-logged-in
users.
If I am right, order conversions will increase
overall with minimal impact to new account
sign-up rates.
Primary Test KPI
Conversion to Order
Data Story Metrics
New Account Creation
Problem
Currently, new customers can create
a new account either on the My
Account interstitial page, or on the
Order Confirmation page. I believe
having the “New Customers” section
at the beginning of the checkout funnel
is overwhelming to visitors overall, as
they have to decide between logging
into an account, checking out as a
guest, and registering a new account.
reducing checkout friction
so let’s givesome real examples
SPRING INTO ACTION
11. surface product options earlier
so let’s givesome real examples
Primary Test KPI
Conversion to Order
Data Story Metrics
PLP Bounce Rate
Problem
Previously, we have tried showing
every color as its own product on the
PLP grid (thus creating many PLPs to
scroll through) or shown only the
“parent product” in a single color on
the grid (thus requiring people to click
into PDPs for more color selections).
Neither solutions have been
satisfactory for our business
of our customers.
I believe that if we denote which
additional colors are available for each
“parent product” on PLPs, customers
will be more likely to identify a product
color selection that appeals to them
earlier in the funnel.
If I am right, we will get more people to
PDPs and see increased conversion rates.
SPRING INTO ACTION
12. I believe that if we offer ShopRunner’s
(SR) free two-day shippingservice to
visitors, then we will increase the
incentive to buy when able to get
items faster at no additional cost.
If I am right, we will see an increase in
overall order conversion rates and revenue,
driven by users taking advantage of the
ShopRunner option.
testing saas benefit claim
so let’s givesome real examples
Primary Test KPI
Conversion to Order
Data Story Metrics
Bounce at Point of
ShopRunner Option
Problem
Currently, we do not have
ShopRunner as a shipping optionon
the site. We offer free 4-7 day
shipping on all orders (no threshold).
Visitors can upgrade to faster
shipping, but it comes as an
additional cost.
SPRING INTO ACTION
13. I believe if we placed an inline email
signup in the header of our desktop
site, we will materially increase email
signups.
If I am right, we will increase email signups
and no decrease in conversion rate.
inline email sign-up
so let’s givesome real examples
Primary Test KPI
Email Newsletter Sign-Up Rate
Data Story Metrics
Conversion to Order
Problem
Currently, we have links to sign-up for
newsletters at the header & footer of
our websites. These links go to a
second page with several forms to fill
in. The extra steps, we believe, create
friction for something that can/should
be very simple.
SPRING INTO ACTION
14. test idea process x
SPRING INTO ACTION
goals & problems
established
Do start with the critical
problems you want to solve
for your customers, mapped
to the measurable goals that
define success.
Do not sneak solutions into
your goal or problem
statements.
hypothesis generation hypothesis prioritization
customer service data
video session recordings
drop off analysis
web analytics
customer service feedback
customer survey data
1
2
3
4
3
1
2
effort to test
potential impact
likelihood belief is true
ability to adopt
and more
X
15. key
takeaways
The magic of optimization & personalization is not simply
creating experiments or campaigns. It is the magic of solving a
customer experience or business problem. Metrics rise in
correlation to your ability to solve problems.
1
Don’t lead with solutions. Don’t chase ideas simply
because others have succeeded with it or because you
are able to test the change. Prioritize your experiments
based, initially, on the problems you expect them to solve.
2
Develop a unifying problem & solution mapping
framework to ensure that you are working in concert to
solve the biggest problems and that you are not
dabbling. Getting winning tests is not winning if the biggest
problems go unsolved and the biggest hypotheses unvalidated.
3
SPRING INTO ACTION