SlideShare a Scribd company logo
1 of 16
Download to read offline
Presentation Name
Speaker’s Name
Title
Ecommerce Test Ideation: What to Optimize and Why
• Optimizely's top solutions partner for enterprise brands & retailers
• Partner to over 30 clients including Adidas, Tory Burch, BluDot,
Whole Foods, Cost Plus World Market, Express, Patagonia &
many more
• Proud winner of 2015 Opticon Partner of the Year & Most
Dramatic Business Impact
hi. i’m matty.
founder & ceo at
SPRING INTO ACTION
Matty Wishnow
& Leo Wishnow
• 15+ years testing & validating as:
• SVP, Direct to Consumer at
Warner Music Group
• Founder & CEO of Insound.com
• Loves “Less and More” design
principles & advanced baseball stats
• Hates dabbling
so, you want
test ideas.
hypotheses learnings investmentsproblemsgoals
wouldn’t you rather a repeatable framework for
producing & prioritizing great test ideas?
SPRING INTO ACTION
the ideas will come.
but, let’s start with the framework.
goals problems hypotheses learning
UX
product
merchandising
marketing
prioritized by data validated by data
investments
1
2
3
Ps
pS
ps
PS
2
3
4
1
2
3
4
1
SPRING INTO ACTION
unifying.
wait like this applies to everything?
#1 Everything is an experiment
Banner/hero changes for new
seasons/campaigns.
Merchandising changes.
Copy changes.
Redesigns & replatforms.
New SaaS additions.
Everything.
#2 Questions to live by:
What problems are the
changes solving?
How will you know if the
changewas successful?
SPRING INTO ACTION
how to
communicate goals
good example:
In 2016, we want to increase
smart phone purchase
conversion rates by 10% y/o/y.
bad example:
We want to improve the mobile
user experience.
Ask yourself:
• Are they widely understood?
• Are they actionable?
• Are they measurable?
• Do they have refined baselines/targets?
• Are they time-based?
SPRING INTO ACTION
how to
capture problems
good example:
Our customers have a problem
understanding the difference
between and value proposition
of products on PLPs.
bad example:
I think we need a responsive
website.
Ask yourself:
• Why is it a problem?
• How does the problem impact the
business or end user?
• Is it a root problem?
• Do you have data to confirm that it is
a significant problem?
SPRING INTO ACTION
how to
articulate a hypothesis
good example:
I believe that if we add geo-targeted
shipping commitments on the shipping
page of checkout, customers will have
more confidence in ordering from us.
If I am right we will see increased conversion
& reduced abandonment at the shipping page
of checkout.
bad example:
I think we need to redesign
checkout.
Try to:
• Be specific about the change you are
contemplating.
• Describe what you expect the result of
that change to be.
• Clearly define the metric you can most
confidently measure success by.
• Start with:
• I believe that…
• If I am right then…
SPRING INTO ACTION
Be consistent about requiring
all changes get mapped to
problems & hypotheses.
2
what do you do once
it’s all mapped?
Determine if additional
problem research is needed.
1
hypotheses learnings investmentsproblemsgoals
Determine what should be
tested vs “just do it”.
3
Review & iterate regulary.4
SPRING INTO ACTION
I believe that if we remove the “New
Customers” section on the interstitial
page between the ShoppingBag and
checkout, we will reduce friction to
continue to checkout for non-logged-in
users.
If I am right, order conversions will increase
overall with minimal impact to new account
sign-up rates.
Primary Test KPI
Conversion to Order
Data Story Metrics
New Account Creation
Problem
Currently, new customers can create
a new account either on the My
Account interstitial page, or on the
Order Confirmation page. I believe
having the “New Customers” section
at the beginning of the checkout funnel
is overwhelming to visitors overall, as
they have to decide between logging
into an account, checking out as a
guest, and registering a new account.
reducing checkout friction
so let’s givesome real examples
SPRING INTO ACTION
surface product options earlier
so let’s givesome real examples
Primary Test KPI
Conversion to Order
Data Story Metrics
PLP Bounce Rate
Problem
Previously, we have tried showing
every color as its own product on the
PLP grid (thus creating many PLPs to
scroll through) or shown only the
“parent product” in a single color on
the grid (thus requiring people to click
into PDPs for more color selections).
Neither solutions have been
satisfactory for our business
of our customers.
I believe that if we denote which
additional colors are available for each
“parent product” on PLPs, customers
will be more likely to identify a product
color selection that appeals to them
earlier in the funnel.
If I am right, we will get more people to
PDPs and see increased conversion rates.
SPRING INTO ACTION
I believe that if we offer ShopRunner’s
(SR) free two-day shippingservice to
visitors, then we will increase the
incentive to buy when able to get
items faster at no additional cost.
If I am right, we will see an increase in
overall order conversion rates and revenue,
driven by users taking advantage of the
ShopRunner option.
testing saas benefit claim
so let’s givesome real examples
Primary Test KPI
Conversion to Order
Data Story Metrics
Bounce at Point of
ShopRunner Option
Problem
Currently, we do not have
ShopRunner as a shipping optionon
the site. We offer free 4-7 day
shipping on all orders (no threshold).
Visitors can upgrade to faster
shipping, but it comes as an
additional cost.
SPRING INTO ACTION
I believe if we placed an inline email
signup in the header of our desktop
site, we will materially increase email
signups.
If I am right, we will increase email signups
and no decrease in conversion rate.
inline email sign-up
so let’s givesome real examples
Primary Test KPI
Email Newsletter Sign-Up Rate
Data Story Metrics
Conversion to Order
Problem
Currently, we have links to sign-up for
newsletters at the header & footer of
our websites. These links go to a
second page with several forms to fill
in. The extra steps, we believe, create
friction for something that can/should
be very simple.
SPRING INTO ACTION
test idea process x
SPRING INTO ACTION
goals & problems
established
Do start with the critical
problems you want to solve
for your customers, mapped
to the measurable goals that
define success.
Do not sneak solutions into
your goal or problem
statements.
hypothesis generation hypothesis prioritization
customer service data
video session recordings
drop off analysis
web analytics
customer service feedback
customer survey data
1
2
3
4
3
1
2
effort to test
potential impact
likelihood belief is true
ability to adopt
and more
X
key
takeaways
The magic of optimization & personalization is not simply
creating experiments or campaigns. It is the magic of solving a
customer experience or business problem. Metrics rise in
correlation to your ability to solve problems.
1
Don’t lead with solutions. Don’t chase ideas simply
because others have succeeded with it or because you
are able to test the change. Prioritize your experiments
based, initially, on the problems you expect them to solve.
2
Develop a unifying problem & solution mapping
framework to ensure that you are working in concert to
solve the biggest problems and that you are not
dabbling. Getting winning tests is not winning if the biggest
problems go unsolved and the biggest hypotheses unvalidated.
3
SPRING INTO ACTION
pages.optimizely.com/spring-into-action
Upcoming Spring Into Action Events
-webinar-series
I

More Related Content

What's hot

Building Stuff To Help You Sell The Stuff You Build
Building Stuff To Help You Sell The Stuff You BuildBuilding Stuff To Help You Sell The Stuff You Build
Building Stuff To Help You Sell The Stuff You BuildPatrick McKenzie
 
Jw conversion elite talk final black
Jw conversion elite talk final blackJw conversion elite talk final black
Jw conversion elite talk final blackAndy Brown
 
Opticon 2015-From Skeptic to Champion
Opticon 2015-From Skeptic to ChampionOpticon 2015-From Skeptic to Champion
Opticon 2015-From Skeptic to ChampionOptimizely
 
Opticon 2015-Optimization at Optimizely
Opticon 2015-Optimization at OptimizelyOpticon 2015-Optimization at Optimizely
Opticon 2015-Optimization at OptimizelyOptimizely
 
Opticon 2017 Do the Thing That Makes the Money
Opticon 2017 Do the Thing That Makes the MoneyOpticon 2017 Do the Thing That Makes the Money
Opticon 2017 Do the Thing That Makes the MoneyOptimizely
 
47 Conversion Rate Optimization Tips
47 Conversion Rate Optimization Tips47 Conversion Rate Optimization Tips
47 Conversion Rate Optimization TipsBrightEdge
 
Optimization Summer Games - Get started with A/B testing
Optimization Summer Games  - Get started with A/B testing Optimization Summer Games  - Get started with A/B testing
Optimization Summer Games - Get started with A/B testing Optimizely
 
Opticon 2015-Scaling Your Testing Program for Maximum Impact
Opticon 2015-Scaling Your Testing Program for Maximum ImpactOpticon 2015-Scaling Your Testing Program for Maximum Impact
Opticon 2015-Scaling Your Testing Program for Maximum ImpactOptimizely
 
A/B Mythbusters: Common Optimization Objections Debunked
A/B Mythbusters: Common Optimization Objections DebunkedA/B Mythbusters: Common Optimization Objections Debunked
A/B Mythbusters: Common Optimization Objections DebunkedOptimizely
 
ABM in Practice: How to Personalize your Homepage
ABM in Practice: How to Personalize your HomepageABM in Practice: How to Personalize your Homepage
ABM in Practice: How to Personalize your HomepageOptimizely
 
Microconf Europe 2013 -- Patrick McKenzie
Microconf Europe 2013 -- Patrick McKenzieMicroconf Europe 2013 -- Patrick McKenzie
Microconf Europe 2013 -- Patrick McKenziePatrick McKenzie
 
UX Analytics and Experimentation for eCommerce Growth
UX Analytics and Experimentation for eCommerce GrowthUX Analytics and Experimentation for eCommerce Growth
UX Analytics and Experimentation for eCommerce GrowthVWO
 
How to Successfully Run Your First Website A/B Test
How to Successfully Run Your First Website A/B TestHow to Successfully Run Your First Website A/B Test
How to Successfully Run Your First Website A/B TestKissmetrics on SlideShare
 
Patrick Campbell — How to Build Actual Customer-Driven Product (Turing Festiv...
Patrick Campbell — How to Build Actual Customer-Driven Product (Turing Festiv...Patrick Campbell — How to Build Actual Customer-Driven Product (Turing Festiv...
Patrick Campbell — How to Build Actual Customer-Driven Product (Turing Festiv...Turing Fest
 
[Elite Camp 2016] Marie Polli - Taking Your Customers on a Journey That Match...
[Elite Camp 2016] Marie Polli - Taking Your Customers on a Journey That Match...[Elite Camp 2016] Marie Polli - Taking Your Customers on a Journey That Match...
[Elite Camp 2016] Marie Polli - Taking Your Customers on a Journey That Match...CXL
 
Kate Moore - Acquisition the Agile Way: Scaling Marketing from $5M to $25M
Kate Moore - Acquisition the Agile Way: Scaling Marketing from $5M to $25MKate Moore - Acquisition the Agile Way: Scaling Marketing from $5M to $25M
Kate Moore - Acquisition the Agile Way: Scaling Marketing from $5M to $25MPrice Intelligently
 
Opticon 2017 Bend the B2B Curve
Opticon 2017 Bend the B2B CurveOpticon 2017 Bend the B2B Curve
Opticon 2017 Bend the B2B CurveOptimizely
 
Opticon 2015-Early Lessons in Personalization Strategy
Opticon 2015-Early Lessons in Personalization StrategyOpticon 2015-Early Lessons in Personalization Strategy
Opticon 2015-Early Lessons in Personalization StrategyOptimizely
 
[Elite Camp 2016] Peep Laja - Fresh Out Of the Oven
[Elite Camp 2016] Peep Laja - Fresh Out Of the Oven[Elite Camp 2016] Peep Laja - Fresh Out Of the Oven
[Elite Camp 2016] Peep Laja - Fresh Out Of the OvenCXL
 

What's hot (20)

Building Stuff To Help You Sell The Stuff You Build
Building Stuff To Help You Sell The Stuff You BuildBuilding Stuff To Help You Sell The Stuff You Build
Building Stuff To Help You Sell The Stuff You Build
 
Jw conversion elite talk final black
Jw conversion elite talk final blackJw conversion elite talk final black
Jw conversion elite talk final black
 
Opticon 2015-From Skeptic to Champion
Opticon 2015-From Skeptic to ChampionOpticon 2015-From Skeptic to Champion
Opticon 2015-From Skeptic to Champion
 
The Power of A/B Testing
The Power of A/B TestingThe Power of A/B Testing
The Power of A/B Testing
 
Opticon 2015-Optimization at Optimizely
Opticon 2015-Optimization at OptimizelyOpticon 2015-Optimization at Optimizely
Opticon 2015-Optimization at Optimizely
 
Opticon 2017 Do the Thing That Makes the Money
Opticon 2017 Do the Thing That Makes the MoneyOpticon 2017 Do the Thing That Makes the Money
Opticon 2017 Do the Thing That Makes the Money
 
47 Conversion Rate Optimization Tips
47 Conversion Rate Optimization Tips47 Conversion Rate Optimization Tips
47 Conversion Rate Optimization Tips
 
Optimization Summer Games - Get started with A/B testing
Optimization Summer Games  - Get started with A/B testing Optimization Summer Games  - Get started with A/B testing
Optimization Summer Games - Get started with A/B testing
 
Opticon 2015-Scaling Your Testing Program for Maximum Impact
Opticon 2015-Scaling Your Testing Program for Maximum ImpactOpticon 2015-Scaling Your Testing Program for Maximum Impact
Opticon 2015-Scaling Your Testing Program for Maximum Impact
 
A/B Mythbusters: Common Optimization Objections Debunked
A/B Mythbusters: Common Optimization Objections DebunkedA/B Mythbusters: Common Optimization Objections Debunked
A/B Mythbusters: Common Optimization Objections Debunked
 
ABM in Practice: How to Personalize your Homepage
ABM in Practice: How to Personalize your HomepageABM in Practice: How to Personalize your Homepage
ABM in Practice: How to Personalize your Homepage
 
Microconf Europe 2013 -- Patrick McKenzie
Microconf Europe 2013 -- Patrick McKenzieMicroconf Europe 2013 -- Patrick McKenzie
Microconf Europe 2013 -- Patrick McKenzie
 
UX Analytics and Experimentation for eCommerce Growth
UX Analytics and Experimentation for eCommerce GrowthUX Analytics and Experimentation for eCommerce Growth
UX Analytics and Experimentation for eCommerce Growth
 
How to Successfully Run Your First Website A/B Test
How to Successfully Run Your First Website A/B TestHow to Successfully Run Your First Website A/B Test
How to Successfully Run Your First Website A/B Test
 
Patrick Campbell — How to Build Actual Customer-Driven Product (Turing Festiv...
Patrick Campbell — How to Build Actual Customer-Driven Product (Turing Festiv...Patrick Campbell — How to Build Actual Customer-Driven Product (Turing Festiv...
Patrick Campbell — How to Build Actual Customer-Driven Product (Turing Festiv...
 
[Elite Camp 2016] Marie Polli - Taking Your Customers on a Journey That Match...
[Elite Camp 2016] Marie Polli - Taking Your Customers on a Journey That Match...[Elite Camp 2016] Marie Polli - Taking Your Customers on a Journey That Match...
[Elite Camp 2016] Marie Polli - Taking Your Customers on a Journey That Match...
 
Kate Moore - Acquisition the Agile Way: Scaling Marketing from $5M to $25M
Kate Moore - Acquisition the Agile Way: Scaling Marketing from $5M to $25MKate Moore - Acquisition the Agile Way: Scaling Marketing from $5M to $25M
Kate Moore - Acquisition the Agile Way: Scaling Marketing from $5M to $25M
 
Opticon 2017 Bend the B2B Curve
Opticon 2017 Bend the B2B CurveOpticon 2017 Bend the B2B Curve
Opticon 2017 Bend the B2B Curve
 
Opticon 2015-Early Lessons in Personalization Strategy
Opticon 2015-Early Lessons in Personalization StrategyOpticon 2015-Early Lessons in Personalization Strategy
Opticon 2015-Early Lessons in Personalization Strategy
 
[Elite Camp 2016] Peep Laja - Fresh Out Of the Oven
[Elite Camp 2016] Peep Laja - Fresh Out Of the Oven[Elite Camp 2016] Peep Laja - Fresh Out Of the Oven
[Elite Camp 2016] Peep Laja - Fresh Out Of the Oven
 

Similar to Ecommerce Test Ideation: What to Optimize and Why

12 meo tang ty le chuyen doi tren trang landingpage
12 meo tang ty le chuyen doi tren trang landingpage12 meo tang ty le chuyen doi tren trang landingpage
12 meo tang ty le chuyen doi tren trang landingpageGetfly CRM
 
Optimizing your marketing automation program for success
Optimizing your marketing automation program for successOptimizing your marketing automation program for success
Optimizing your marketing automation program for successGetResponse
 
Driving Product Success through Experimentation
Driving Product Success through ExperimentationDriving Product Success through Experimentation
Driving Product Success through ExperimentationNUS-ISS
 
Ten Powerful Tips To Get More From Your Analytics
Ten Powerful Tips To Get More From Your AnalyticsTen Powerful Tips To Get More From Your Analytics
Ten Powerful Tips To Get More From Your AnalyticsMichele Kiss
 
5 Ways to Increase B2B SaaS Revenue
5 Ways to Increase B2B SaaS Revenue5 Ways to Increase B2B SaaS Revenue
5 Ways to Increase B2B SaaS Revenuestimulead
 
UX Design Workshop - Tableau CRM Edition
UX Design Workshop - Tableau CRM EditionUX Design Workshop - Tableau CRM Edition
UX Design Workshop - Tableau CRM Editionrikkehovgaard
 
Innovators canvas template
Innovators canvas templateInnovators canvas template
Innovators canvas templateJake Nielson
 
Innovators canvas template
Innovators canvas templateInnovators canvas template
Innovators canvas templateJake Nielson
 
Trade up from cube to corner office - OpenText TeamSite Optimost
Trade up from cube to corner office - OpenText TeamSite OptimostTrade up from cube to corner office - OpenText TeamSite Optimost
Trade up from cube to corner office - OpenText TeamSite Optimostsuparupaa
 
Trade up from cube to corner office - OpenText Optimost
Trade up from cube to corner office - OpenText OptimostTrade up from cube to corner office - OpenText Optimost
Trade up from cube to corner office - OpenText OptimostOpenText
 
Proactivep rospecting
Proactivep rospectingProactivep rospecting
Proactivep rospectingSAFDAR RASOOL
 
Rapid Growth Driven by Analytics
Rapid Growth Driven by AnalyticsRapid Growth Driven by Analytics
Rapid Growth Driven by AnalyticsSujan Patel
 
An introduction to NPS - Net Promoter Score
An introduction to NPS - Net Promoter ScoreAn introduction to NPS - Net Promoter Score
An introduction to NPS - Net Promoter ScoreStartquestion
 
Pursuing Customer Inspired Growth
Pursuing Customer Inspired GrowthPursuing Customer Inspired Growth
Pursuing Customer Inspired GrowthKearney
 
The value proposition builder workbook
The value proposition builder workbookThe value proposition builder workbook
The value proposition builder workbookIncubation & Industry
 
2017 06-test withintelligence-conversionelite
2017 06-test withintelligence-conversionelite2017 06-test withintelligence-conversionelite
2017 06-test withintelligence-conversioneliteTim Stewart
 
Why We Test - Rethinking Your Approach
Why We Test - Rethinking Your ApproachWhy We Test - Rethinking Your Approach
Why We Test - Rethinking Your Approachaudreybloemer
 

Similar to Ecommerce Test Ideation: What to Optimize and Why (20)

12 meo tang ty le chuyen doi tren trang landingpage
12 meo tang ty le chuyen doi tren trang landingpage12 meo tang ty le chuyen doi tren trang landingpage
12 meo tang ty le chuyen doi tren trang landingpage
 
Optimizing your marketing automation program for success
Optimizing your marketing automation program for successOptimizing your marketing automation program for success
Optimizing your marketing automation program for success
 
Driving Product Success through Experimentation
Driving Product Success through ExperimentationDriving Product Success through Experimentation
Driving Product Success through Experimentation
 
Ten Powerful Tips To Get More From Your Analytics
Ten Powerful Tips To Get More From Your AnalyticsTen Powerful Tips To Get More From Your Analytics
Ten Powerful Tips To Get More From Your Analytics
 
5 Ways to Increase B2B SaaS Revenue
5 Ways to Increase B2B SaaS Revenue5 Ways to Increase B2B SaaS Revenue
5 Ways to Increase B2B SaaS Revenue
 
UX Design Workshop - Tableau CRM Edition
UX Design Workshop - Tableau CRM EditionUX Design Workshop - Tableau CRM Edition
UX Design Workshop - Tableau CRM Edition
 
Innovators canvas template
Innovators canvas templateInnovators canvas template
Innovators canvas template
 
Innovators canvas template
Innovators canvas templateInnovators canvas template
Innovators canvas template
 
Trade up from cube to corner office - OpenText TeamSite Optimost
Trade up from cube to corner office - OpenText TeamSite OptimostTrade up from cube to corner office - OpenText TeamSite Optimost
Trade up from cube to corner office - OpenText TeamSite Optimost
 
Trade up from cube to corner office - OpenText Optimost
Trade up from cube to corner office - OpenText OptimostTrade up from cube to corner office - OpenText Optimost
Trade up from cube to corner office - OpenText Optimost
 
Vision Crafting
Vision Crafting Vision Crafting
Vision Crafting
 
Proactivep rospecting
Proactivep rospectingProactivep rospecting
Proactivep rospecting
 
Rapid Growth Driven by Analytics
Rapid Growth Driven by AnalyticsRapid Growth Driven by Analytics
Rapid Growth Driven by Analytics
 
An introduction to NPS - Net Promoter Score
An introduction to NPS - Net Promoter ScoreAn introduction to NPS - Net Promoter Score
An introduction to NPS - Net Promoter Score
 
Pursuing Customer Inspired Growth
Pursuing Customer Inspired GrowthPursuing Customer Inspired Growth
Pursuing Customer Inspired Growth
 
The value proposition builder workbook
The value proposition builder workbookThe value proposition builder workbook
The value proposition builder workbook
 
Example buyer persona marketer
Example buyer persona marketer Example buyer persona marketer
Example buyer persona marketer
 
2017 06-test withintelligence-conversionelite
2017 06-test withintelligence-conversionelite2017 06-test withintelligence-conversionelite
2017 06-test withintelligence-conversionelite
 
Ab testing for_beginners_
Ab testing for_beginners_Ab testing for_beginners_
Ab testing for_beginners_
 
Why We Test - Rethinking Your Approach
Why We Test - Rethinking Your ApproachWhy We Test - Rethinking Your Approach
Why We Test - Rethinking Your Approach
 

More from Optimizely

Clover Rings Up Digital Growth to Drive Experimentation
Clover Rings Up Digital Growth to Drive ExperimentationClover Rings Up Digital Growth to Drive Experimentation
Clover Rings Up Digital Growth to Drive ExperimentationOptimizely
 
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...Optimizely
 
The Science of Getting Testing Right
The Science of Getting Testing RightThe Science of Getting Testing Right
The Science of Getting Testing RightOptimizely
 
Atlassian's Mystique CLI, Minimizing the Experiment Development Cycle
Atlassian's Mystique CLI, Minimizing the Experiment Development CycleAtlassian's Mystique CLI, Minimizing the Experiment Development Cycle
Atlassian's Mystique CLI, Minimizing the Experiment Development CycleOptimizely
 
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...Optimizely
 
Zillow + Optimizely: Building the Bridge to $20 Billion Revenue
Zillow + Optimizely: Building the Bridge to $20 Billion RevenueZillow + Optimizely: Building the Bridge to $20 Billion Revenue
Zillow + Optimizely: Building the Bridge to $20 Billion RevenueOptimizely
 
The Future of Optimizely for Technical Teams
The Future of Optimizely for Technical TeamsThe Future of Optimizely for Technical Teams
The Future of Optimizely for Technical TeamsOptimizely
 
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...Optimizely
 
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...Optimizely
 
Building an Experiment Pipeline for GitHub’s New Free Team Offering
Building an Experiment Pipeline for GitHub’s New Free Team OfferingBuilding an Experiment Pipeline for GitHub’s New Free Team Offering
Building an Experiment Pipeline for GitHub’s New Free Team OfferingOptimizely
 
AMC Networks Experiments Faster on the Server Side
AMC Networks Experiments Faster on the Server SideAMC Networks Experiments Faster on the Server Side
AMC Networks Experiments Faster on the Server SideOptimizely
 
Evolving Experimentation from CRO to Product Development
Evolving Experimentation from CRO to Product DevelopmentEvolving Experimentation from CRO to Product Development
Evolving Experimentation from CRO to Product DevelopmentOptimizely
 
Overcoming the Challenges of Experimentation on a Service Oriented Architecture
Overcoming the Challenges of Experimentation on a Service Oriented ArchitectureOvercoming the Challenges of Experimentation on a Service Oriented Architecture
Overcoming the Challenges of Experimentation on a Service Oriented ArchitectureOptimizely
 
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...Optimizely
 
Making Your Hypothesis Work Harder to Inform Future Product Strategy
Making Your Hypothesis Work Harder to Inform Future Product StrategyMaking Your Hypothesis Work Harder to Inform Future Product Strategy
Making Your Hypothesis Work Harder to Inform Future Product StrategyOptimizely
 
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives RevenueKick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives RevenueOptimizely
 
Experimentation through Clients' Eyes
Experimentation through Clients' EyesExperimentation through Clients' Eyes
Experimentation through Clients' EyesOptimizely
 
Shipping to Learn and Accelerate Growth with GitHub
Shipping to Learn and Accelerate Growth with GitHubShipping to Learn and Accelerate Growth with GitHub
Shipping to Learn and Accelerate Growth with GitHubOptimizely
 
Test Everything: TrustRadius Delivers Customer Value with Experimentation
Test Everything: TrustRadius Delivers Customer Value with ExperimentationTest Everything: TrustRadius Delivers Customer Value with Experimentation
Test Everything: TrustRadius Delivers Customer Value with ExperimentationOptimizely
 
Optimizely Agent: Scaling Resilient Feature Delivery
Optimizely Agent: Scaling Resilient Feature DeliveryOptimizely Agent: Scaling Resilient Feature Delivery
Optimizely Agent: Scaling Resilient Feature DeliveryOptimizely
 

More from Optimizely (20)

Clover Rings Up Digital Growth to Drive Experimentation
Clover Rings Up Digital Growth to Drive ExperimentationClover Rings Up Digital Growth to Drive Experimentation
Clover Rings Up Digital Growth to Drive Experimentation
 
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
 
The Science of Getting Testing Right
The Science of Getting Testing RightThe Science of Getting Testing Right
The Science of Getting Testing Right
 
Atlassian's Mystique CLI, Minimizing the Experiment Development Cycle
Atlassian's Mystique CLI, Minimizing the Experiment Development CycleAtlassian's Mystique CLI, Minimizing the Experiment Development Cycle
Atlassian's Mystique CLI, Minimizing the Experiment Development Cycle
 
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
 
Zillow + Optimizely: Building the Bridge to $20 Billion Revenue
Zillow + Optimizely: Building the Bridge to $20 Billion RevenueZillow + Optimizely: Building the Bridge to $20 Billion Revenue
Zillow + Optimizely: Building the Bridge to $20 Billion Revenue
 
The Future of Optimizely for Technical Teams
The Future of Optimizely for Technical TeamsThe Future of Optimizely for Technical Teams
The Future of Optimizely for Technical Teams
 
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
 
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
 
Building an Experiment Pipeline for GitHub’s New Free Team Offering
Building an Experiment Pipeline for GitHub’s New Free Team OfferingBuilding an Experiment Pipeline for GitHub’s New Free Team Offering
Building an Experiment Pipeline for GitHub’s New Free Team Offering
 
AMC Networks Experiments Faster on the Server Side
AMC Networks Experiments Faster on the Server SideAMC Networks Experiments Faster on the Server Side
AMC Networks Experiments Faster on the Server Side
 
Evolving Experimentation from CRO to Product Development
Evolving Experimentation from CRO to Product DevelopmentEvolving Experimentation from CRO to Product Development
Evolving Experimentation from CRO to Product Development
 
Overcoming the Challenges of Experimentation on a Service Oriented Architecture
Overcoming the Challenges of Experimentation on a Service Oriented ArchitectureOvercoming the Challenges of Experimentation on a Service Oriented Architecture
Overcoming the Challenges of Experimentation on a Service Oriented Architecture
 
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
 
Making Your Hypothesis Work Harder to Inform Future Product Strategy
Making Your Hypothesis Work Harder to Inform Future Product StrategyMaking Your Hypothesis Work Harder to Inform Future Product Strategy
Making Your Hypothesis Work Harder to Inform Future Product Strategy
 
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives RevenueKick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
 
Experimentation through Clients' Eyes
Experimentation through Clients' EyesExperimentation through Clients' Eyes
Experimentation through Clients' Eyes
 
Shipping to Learn and Accelerate Growth with GitHub
Shipping to Learn and Accelerate Growth with GitHubShipping to Learn and Accelerate Growth with GitHub
Shipping to Learn and Accelerate Growth with GitHub
 
Test Everything: TrustRadius Delivers Customer Value with Experimentation
Test Everything: TrustRadius Delivers Customer Value with ExperimentationTest Everything: TrustRadius Delivers Customer Value with Experimentation
Test Everything: TrustRadius Delivers Customer Value with Experimentation
 
Optimizely Agent: Scaling Resilient Feature Delivery
Optimizely Agent: Scaling Resilient Feature DeliveryOptimizely Agent: Scaling Resilient Feature Delivery
Optimizely Agent: Scaling Resilient Feature Delivery
 

Recently uploaded

Retail OS - Delivering the Omnichannel Experience.pptx
Retail OS - Delivering the Omnichannel Experience.pptxRetail OS - Delivering the Omnichannel Experience.pptx
Retail OS - Delivering the Omnichannel Experience.pptxJohn Andrews
 
Aliexpress Coupon Codes And Discount Codes
Aliexpress Coupon Codes And Discount CodesAliexpress Coupon Codes And Discount Codes
Aliexpress Coupon Codes And Discount Codesbirs gonzo
 
Key-Benefits-of-Marketplace-Payment-Reconciliation
Key-Benefits-of-Marketplace-Payment-ReconciliationKey-Benefits-of-Marketplace-Payment-Reconciliation
Key-Benefits-of-Marketplace-Payment-ReconciliationVinculum Solutions Pvt. Ltd.
 
Delihvery-Delivery Partner-Ecommerce Platform
Delihvery-Delivery Partner-Ecommerce PlatformDelihvery-Delivery Partner-Ecommerce Platform
Delihvery-Delivery Partner-Ecommerce PlatformDeborahnich
 
Web 3 in Retail Unlocking New Possibilities
Web 3 in Retail Unlocking New PossibilitiesWeb 3 in Retail Unlocking New Possibilities
Web 3 in Retail Unlocking New PossibilitiesLiveplex
 

Recently uploaded (6)

Ball Pens
Ball                                PensBall                                Pens
Ball Pens
 
Retail OS - Delivering the Omnichannel Experience.pptx
Retail OS - Delivering the Omnichannel Experience.pptxRetail OS - Delivering the Omnichannel Experience.pptx
Retail OS - Delivering the Omnichannel Experience.pptx
 
Aliexpress Coupon Codes And Discount Codes
Aliexpress Coupon Codes And Discount CodesAliexpress Coupon Codes And Discount Codes
Aliexpress Coupon Codes And Discount Codes
 
Key-Benefits-of-Marketplace-Payment-Reconciliation
Key-Benefits-of-Marketplace-Payment-ReconciliationKey-Benefits-of-Marketplace-Payment-Reconciliation
Key-Benefits-of-Marketplace-Payment-Reconciliation
 
Delihvery-Delivery Partner-Ecommerce Platform
Delihvery-Delivery Partner-Ecommerce PlatformDelihvery-Delivery Partner-Ecommerce Platform
Delihvery-Delivery Partner-Ecommerce Platform
 
Web 3 in Retail Unlocking New Possibilities
Web 3 in Retail Unlocking New PossibilitiesWeb 3 in Retail Unlocking New Possibilities
Web 3 in Retail Unlocking New Possibilities
 

Ecommerce Test Ideation: What to Optimize and Why

  • 1. Presentation Name Speaker’s Name Title Ecommerce Test Ideation: What to Optimize and Why
  • 2. • Optimizely's top solutions partner for enterprise brands & retailers • Partner to over 30 clients including Adidas, Tory Burch, BluDot, Whole Foods, Cost Plus World Market, Express, Patagonia & many more • Proud winner of 2015 Opticon Partner of the Year & Most Dramatic Business Impact hi. i’m matty. founder & ceo at SPRING INTO ACTION Matty Wishnow & Leo Wishnow • 15+ years testing & validating as: • SVP, Direct to Consumer at Warner Music Group • Founder & CEO of Insound.com • Loves “Less and More” design principles & advanced baseball stats • Hates dabbling
  • 3. so, you want test ideas. hypotheses learnings investmentsproblemsgoals wouldn’t you rather a repeatable framework for producing & prioritizing great test ideas? SPRING INTO ACTION
  • 4. the ideas will come. but, let’s start with the framework. goals problems hypotheses learning UX product merchandising marketing prioritized by data validated by data investments 1 2 3 Ps pS ps PS 2 3 4 1 2 3 4 1 SPRING INTO ACTION
  • 5. unifying. wait like this applies to everything? #1 Everything is an experiment Banner/hero changes for new seasons/campaigns. Merchandising changes. Copy changes. Redesigns & replatforms. New SaaS additions. Everything. #2 Questions to live by: What problems are the changes solving? How will you know if the changewas successful? SPRING INTO ACTION
  • 6. how to communicate goals good example: In 2016, we want to increase smart phone purchase conversion rates by 10% y/o/y. bad example: We want to improve the mobile user experience. Ask yourself: • Are they widely understood? • Are they actionable? • Are they measurable? • Do they have refined baselines/targets? • Are they time-based? SPRING INTO ACTION
  • 7. how to capture problems good example: Our customers have a problem understanding the difference between and value proposition of products on PLPs. bad example: I think we need a responsive website. Ask yourself: • Why is it a problem? • How does the problem impact the business or end user? • Is it a root problem? • Do you have data to confirm that it is a significant problem? SPRING INTO ACTION
  • 8. how to articulate a hypothesis good example: I believe that if we add geo-targeted shipping commitments on the shipping page of checkout, customers will have more confidence in ordering from us. If I am right we will see increased conversion & reduced abandonment at the shipping page of checkout. bad example: I think we need to redesign checkout. Try to: • Be specific about the change you are contemplating. • Describe what you expect the result of that change to be. • Clearly define the metric you can most confidently measure success by. • Start with: • I believe that… • If I am right then… SPRING INTO ACTION
  • 9. Be consistent about requiring all changes get mapped to problems & hypotheses. 2 what do you do once it’s all mapped? Determine if additional problem research is needed. 1 hypotheses learnings investmentsproblemsgoals Determine what should be tested vs “just do it”. 3 Review & iterate regulary.4 SPRING INTO ACTION
  • 10. I believe that if we remove the “New Customers” section on the interstitial page between the ShoppingBag and checkout, we will reduce friction to continue to checkout for non-logged-in users. If I am right, order conversions will increase overall with minimal impact to new account sign-up rates. Primary Test KPI Conversion to Order Data Story Metrics New Account Creation Problem Currently, new customers can create a new account either on the My Account interstitial page, or on the Order Confirmation page. I believe having the “New Customers” section at the beginning of the checkout funnel is overwhelming to visitors overall, as they have to decide between logging into an account, checking out as a guest, and registering a new account. reducing checkout friction so let’s givesome real examples SPRING INTO ACTION
  • 11. surface product options earlier so let’s givesome real examples Primary Test KPI Conversion to Order Data Story Metrics PLP Bounce Rate Problem Previously, we have tried showing every color as its own product on the PLP grid (thus creating many PLPs to scroll through) or shown only the “parent product” in a single color on the grid (thus requiring people to click into PDPs for more color selections). Neither solutions have been satisfactory for our business of our customers. I believe that if we denote which additional colors are available for each “parent product” on PLPs, customers will be more likely to identify a product color selection that appeals to them earlier in the funnel. If I am right, we will get more people to PDPs and see increased conversion rates. SPRING INTO ACTION
  • 12. I believe that if we offer ShopRunner’s (SR) free two-day shippingservice to visitors, then we will increase the incentive to buy when able to get items faster at no additional cost. If I am right, we will see an increase in overall order conversion rates and revenue, driven by users taking advantage of the ShopRunner option. testing saas benefit claim so let’s givesome real examples Primary Test KPI Conversion to Order Data Story Metrics Bounce at Point of ShopRunner Option Problem Currently, we do not have ShopRunner as a shipping optionon the site. We offer free 4-7 day shipping on all orders (no threshold). Visitors can upgrade to faster shipping, but it comes as an additional cost. SPRING INTO ACTION
  • 13. I believe if we placed an inline email signup in the header of our desktop site, we will materially increase email signups. If I am right, we will increase email signups and no decrease in conversion rate. inline email sign-up so let’s givesome real examples Primary Test KPI Email Newsletter Sign-Up Rate Data Story Metrics Conversion to Order Problem Currently, we have links to sign-up for newsletters at the header & footer of our websites. These links go to a second page with several forms to fill in. The extra steps, we believe, create friction for something that can/should be very simple. SPRING INTO ACTION
  • 14. test idea process x SPRING INTO ACTION goals & problems established Do start with the critical problems you want to solve for your customers, mapped to the measurable goals that define success. Do not sneak solutions into your goal or problem statements. hypothesis generation hypothesis prioritization customer service data video session recordings drop off analysis web analytics customer service feedback customer survey data 1 2 3 4 3 1 2 effort to test potential impact likelihood belief is true ability to adopt and more X
  • 15. key takeaways The magic of optimization & personalization is not simply creating experiments or campaigns. It is the magic of solving a customer experience or business problem. Metrics rise in correlation to your ability to solve problems. 1 Don’t lead with solutions. Don’t chase ideas simply because others have succeeded with it or because you are able to test the change. Prioritize your experiments based, initially, on the problems you expect them to solve. 2 Develop a unifying problem & solution mapping framework to ensure that you are working in concert to solve the biggest problems and that you are not dabbling. Getting winning tests is not winning if the biggest problems go unsolved and the biggest hypotheses unvalidated. 3 SPRING INTO ACTION