This document provides a guide for optimizing testing at an e-commerce company through an AB/MVT testing framework. It outlines objectives to improve customer experiences and business metrics through iterative testing. Key recommendations include establishing strategic testing objectives, analyzing and retesting hypotheses to uncover hidden behavioral insights, implementing frequent simple tests, and instituting testing as a core company practice. Challenges may include ensuring adequate resources and coordination across teams.

1. AB/MV Testing
A Guide to Passion Optimization
By Josh Stanley

2. Guide Outline:
• Why Testing Matters
• OKR Framework
• Objectives
• Recommendations

3. Why AB/MVT testing matters?
• AB/MVT is a tool for connecting people to their passion.
Testing allows us to truly create specialty retail sites that best serve
the customer.
• There’s big money in small changes. We saw $x million in
incremental revenue last year from testing, and we only ran xx
major tests.
• Pain points exist. We know pain points exist for our customers
and we can identify the right way to address them.
• We don’t know what really works. On average, 20% of
experiments are winners. So we’re right 1 out 5 times
• Competitive advantage is out there. The opportunity to
understand our customer better and refine our strategy around
what they perceive and care about is huge!

4. Objectives KPIs Risks
“Improve what matters” • Incremental revenue gains of $XX
million.
•
•
Late start
Lack of dedicated resources
Get more people to the products they
• Bounce rates – , Exit pages – • Time for coordination of testing
want, adding more gear to carts, and
• Add to cart +, AOV + with other efforts
checking out more often.
• Cart Abandonment –, Conversion +
“Iterate on lessons” • Increase the # of concurrent
campaigns by X%
•
•
Lack of a repository
Time to coordinate test
Seek to understand behavior behind
• Increase reused campaigns and retrospectives and promote
test results with variations of the
versioning by X% follow-up tests
same test.
“Test early and often” •
•
Increase the # of tests run by X%
Reduce time to test by X%
•
•
Late start
Time to focus on process
Create simple and effective processes
• Full portfolio testing improvement
to test everything, everywhere, and
• Cost effectiveness of scaling our
all the time.
tool
“Institutionalize testing” • Increase user handling of testing
by X%
• Time to campaign, educate, and
propagate testing.
Make AB/MVT testing part of the
• Increase backlog by X%
culture. Bring AB/MVT testing into
the daily workflow of key
stakeholders.

5. “Improve what matters”
Current Process Where’s the focus and strategy alignment?
Competitive Analysis
Data Mining
Best Practices Idea What about What about Is this in
Risk Mitigating exit pages? Customer pain strategic
points? alignment?
Home
Page
Requirements & KPIs
Hypothesis
PLP
PDP
Prediction
Cart
Page
Test Checkout
Page
Success or failure?
“Decide and move on” Result

6. “Improve what matters”
Ideal Process Focus on a strategic objective
Define problem statements
Get “ideas” focused
Strategic Work from a backlog
Objective
(Focus)
Hypothesis
Frame for tests at the start for re-testing
Analyze &
Hypothesis Prediction
Retest
Align expectations to the objective(s)
Test
Building tests to support variations
Analyze & Retest
Test Prediction
Analysis with a retrospective
Retest with new variations, rules, or
thresholds to refine learning.

7. “Iterate on lessons”
Single pass AB/MVT testing does not prove a hypothesis.
It only supports the hypothesis.
So, why do we stop learning after we identify a “winner”?
Why do we assume we tested the hypothesis correctly when we see a “loser”?
THE REAL VALUE PROPOSITION OF TESTING IS THE “HIDDEN VALUE”!
Performance Drivers (Quantitative):
• What reduces bounce rates?
Guaranteed Value
• What drives conversion?
• What increases AOV?
Behavioral Knowledge (Qualitative):
• Do we understand our customer?
Hidden Value
• How do they make decisions?
• Does reality match perception?

8. “Iterate on lessons”
If our goal is to get to the core of our customer’s behavior,
why isn’t behavioral study apart of what we do?
Competitive Testing
Best Practices
Analysis
Data Mining
Cycles Behavior
From the “Consumer
Decisions” class I
taught - based on an
employee survey.
Even Backcountry
employees are
susceptible to
cognitive biases. 

9. “Test early and often”
Our Test Types
Optimization
Changing elements of the site
to achieve “local maximum”
Conceptual
Feature improvements testing
that’s informative, or supports
“better design” hypothesis
Functional
Site elements that should be
dynamic with functional
ownership (i.e. targeting)

10. “Test early and often”
Process Principals for Velocity:
• Follow agile philosophies where we can: backlog, sprints,
retrospectives, etc.
• Incorporate the 3rd party into the agile process.
• Test “small” when it makes sense. Get down to the essence of the
hypothesis to ensure we’re not over-scoping the test.
• Make the process simple and transparent.

11. “Institutionalize testing”
• Functional Testing Online Merchandising
Functional Test Concept
– Identifying the opportunities to
create stakeholder owned
functional tests.
• Educational Campaign
– Tech Series Speakers
– Wisdom Wednesdays
• Testing Campaign
– Workshops, Trainings
– Infographics, Blogs, etc.
• Engineer the Process Allowing stakeholder to
A/B/n test any messaging in
– Build testing into the dev process slots at any time!

12. “Institutionalize testing”
Areas to focus on…
• Test coordination
– Understanding and resolving conflicts
– Transparency into testing priorities and progress
– Communicating campaigns, impacts, and follow-up
• Maintaining the backlog and functional tests
– Prioritizing the backlog based on strategy and predictive value
– Maintaining functional tests and resulting value
– Team BL  Monetate BL  Creative BL  BI BL  SPI BL  Test
• Knowledge management
– Building the knowledge repository/reference guide
– Coordinating knowledge share activities (workshops, retrospectives, etc.)
– Tracking and reporting testing value.
• Continuous Improvement/ Roadmap

13. Recommendation
We need a dedicated Corp
Initiatives
Visual
Merch
Content
UX
“Testing Manager”. Mktg BI
– Product/Project-like role
– Backlog(s) Management
– Department Coordination
Portfolio Testing Backlog
– Vendor Management Creative, BI, and SPI Requests
– Internal Champion
Product Management
Testing Testing Testing
– Future Planning Stories Stories Stories
(CSS, (CSS, Javasc (CSS, Javasc
– Reporting Javascript) ript) ript)
Bike Outdoor ASR
Backlog Backlog Backlog
Product Manager, A/B/n Testing