In this webinar, Hazjier will help you: - Discover common challenges that keep visitors from converting - Understand what tests will actually impact your revenue goals - Learn how other businesses have used A/B testing to drive results

2. Generating High-­Impact Experiments
Hazjier Pourkhalkhali
Senior Strategy Consultant
@Hazjier

3. Optimizely is the world’s largest digital laboratory

5. We researched the world’s top companies
Interviewed experts
§ Spoke with thought
leaders in the industry
§ Met with top customers
Built maturity model
§ Developed
comprehensive
maturity model
§ Assessed 100
customers
§ Surveyed 50+
customers
Held 100+ Business Reviews
§ Met with executives
at over 100
Optimizely clients
Analyzed Top Revenue Experiments
§ Analyzed the 100 best
experiments run on
Revenue Per Visitor
§ Asked customers to
describe test and process

6. User
acquisition:
1.5m
User
retention
33%
Add-to-Cart:
4.00%
Cart
Checkout:
50.0%
Average
quantity:
2.00
Average per
unit price:
€25.00
Average
Order Value:
€50.00
Conversion
Rate:
2.00%
Revenue Per
Visitor:
€1.00
Visitors:
2.0m
Revenue:
€2.0m
Understand the revenue drivers

7. Goals
Strategies
Tactics
Cart
Checkout
Rate
Wording /
messaging
Location /
size
Design /
color /
iconograp
hy
Establish a
visual
hierarchy
Remove
non-
critical
content
Limit
number of
choices
Select/
target the
right USP’s
Refine the
messaging
Crystallize
visually
Clear and
consistent
UI
Structure,
naming,
and order
of sections
Describe
visually
Test ideal
product
visuals
Use the
right
review
systems
Descriptions
Reviews
Images
Emphasize
the Primary
Call to Action
Minimize
Distractions
Communicate
Unique
Selling Points
Design
Intuitive
Navigation
Market
Products
Effectively
Build a Sense
of Urgency
Show
quantity
left
Push
temporary
offers
Promote
buying
before set
times

8. Testing requires freedom
LOW FREEDOM HIGH FREEDOM
“Every change we make to our
website has to be approved
by our senior staff”
-Financial Services company

9. Testing requires freedom
LOW FREEDOM HIGH FREEDOM
“Our Brand Styleguide is key.
We cannot make changes that
do not fit our design book”
-Financial Services company

10. Testing requires freedom
LOW FREEDOM HIGH FREEDOM
“We would change our
company logo if it
converted better”
-Travel Company

12. NO BEARD CAVEMAN
.30% .29% .28% .32% .25% .74%
.11% .28% .24% .32% .09% .59%
.74%
.59%

13. Minimize Distractions:
Remove Non-Critical Content
+14%
Revenue Per
Visitor

14. BASED ON KEYWORDCURATED
+24%
Click to Sale
+30%
Add-to-Cart
+210%
Conversions

17. § The user is reminded
repeatedly of price
§ The user must remember the
product benefits
§ The user must scroll down to
the CTA when their basket is
3 or more
Problem Solution Result
§ Reduce size of price
§ Highlight discounts
§ Add product visuals
§ Duplicate CTA at top of
page
§ Increase Conversion Rate
§ Increase Conversion Rate
§ Increase CTR to checkout
Build powerful hypotheses

18. § Analyze one section of your
application / website
§ Document 10 different
problems for your users
§ Group them by common root
causes
§ Prioritize by urgency
Problem Solution Result
Optimizely Hypothesis Workshop

19. § Analyze one section of your
application / website
§ Document 10 different
problems for your users
§ Group them by common root
causes
§ Prioritize by urgency
Problem Solution Result
Optimizely Hypothesis Workshop
§ Select 1 concrete
problem
§ Design 10 solutions per
theme
§ Choose 4 preferred
solutions

20. The problemwith “A/B Testing”
2
A B
4
A B C D

21. Most people test only 2 variations
0
10,000
20,000
30,000
40,000
50,000
60,000
2 3 4 5 6 7 >8
Tests
65% 17% 17%
# of Variations

22. Most people test only 2 variations
0
10,000
20,000
30,000
40,000
50,000
60,000
2 3 4 5 6 7 >8
Tests
# of Variations

23. Test 5 variationsand nearly double your win rate!
Win Rate
# of Variations
14% 18% 20% 27% 20% 22% 23%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2 3 4 5 6 7 >8
+90%

24. Running more variationswill significantlyimprove your testing!

25. § Analyze one section of your
application / website
§ Document 10 different
problems for your users
§ Group them by common root
causes
§ Prioritize by urgency
Problem Solution Result
Optimizely Hypothesis Workshop
§ Select 1 concrete
problem
§ Design 10 solutions per
theme
§ Choose 4 preferred
solutions
§ Primary Goals:
What do you seek to impact
immediately
§ Secondary Goals:
What would you like to
impact, but do not have to
§ Monitoring Goals:
What are metrics you want
to track

26. Optimizely funnel analysis
Homepage
Flight
Results
Book
Flight
Passenger
Details
Flight
Options
Payment
Page
Step
45%
30%
65% 65%
90%
70%
40%
40%
15%
20%
5%
25%
15%
30%
20%
15%
5% 5%
Exit Rate
Divergence
Completion

27. Optimizely funnel analysis
Homepage
Flight
Results
Book
Flight
Passenger
Details
Flight
Options
Payment
Page
Step
Primary /
Secondary
Goals
§ Searches
Submitted
§ Category tabs
§ Deals
§ Frequent Flyer
§ Flight Selection
§ Filter Usage
§ Refine Search
§ Sort Order
§ Continue
button
§ Edit search
§ Upsells
§ Field
completion
rate
§ Continue
button
§ Continue
button
§ Upsells
§ Seat
selection
§ Conversion
rate
§ Form field
completion
§ Payment
Type
Monitoring
Goals
§ Checkboxes
§ Flight tracker
§ Home page
clicks
§ Menu
interactions
§ Site search
§ Home page
clicks
§ Menu
interactions
§ Site search
§ Field error
rate
§ Home page
clicks
§ Menu
interactions
§ Site search
§ Shopping
cart
pageviews
§ Home page
clicks
§ Return to
previous page
§ Edit address,
information
§ Error rate

28. § Analyze one section of your
application / website
§ Document 10 different
problems for your users
§ Group them by common root
causes
§ Prioritize by urgency
Problem Solution Result
Optimizely Hypothesis Workshop
§ Select 1 concrete
problem
§ Design 10 solutions per
theme
§ Choose 4 preferred
solutions
§ Primary Goals:
What do you seek to impact
immediately
§ Secondary Goals:
What would you like to
impact, but do not have to
§ Monitoring Goals:
What are metrics you want
to track