The document outlines the structure and purpose of a summer school program by Optimizely, focusing on optimizing online business performance through effective A/B testing and hypothesis creation. It provides examples of actionable hypotheses related to enhancing user experience and increasing conversion rates. The goal is to enable participants to gather insights and drive data-driven decisions to achieve specific business and optimization goals.

1. Presentation Title
#OptimizelySchoolwww.summerschool2015.eu

2. Presentation Title
#OptimizelySchoolwww.summerschool2015.eu

3. What is Summer School?
Optimizely Summer School –
Class 1
#OptimizelySchoolwww.summerschool2015.eu

4. Annemarie de Jong
Optimization Consultant
 annemarie@optimizely.com
 @annemariedejong

5. #OptimizelySchoolwww.summerschool2015.eu

6. Class 2
How to create solid hypotheses
and set up powerful A/B tests
#OptimizelySchoolwww.summerschool2015.eu

7. Company Goal
Business Unit
Optimization
Goals
Experiment
Goals
#OptimizelySchool

9. Improve annual profit
by being a reliable go to
store for outdoor
Company Goal
Reduce shipping
& handling costs
from returns
Business Unit
Increase UX of
the order process
Optimization
Goals
Relation size
calculator and UX
Experiment
Goals
#OptimizelySchool

10. Success metrics differ per industry
Ecommerce/ Travel Media Lead Gen/ B2B
Product page views Pages per visit Key content page views
Add to cart rate Ad impressions Content engagement
Checkout completion rate Ad viewability AD unit CTR Funnel entrance rate
Average order value CPM CPC Entry field completion
Conversion rate Subscription rate Conversion rate
#OptimizelySchoolwww.summerschool2015.eu

11.  There is not enough
motivating imagery
 Page is overly focused
on price
 Call to Action is too far
down the page
Problem Solution Result
How to find testing ideas
 Add product visuals
 Balance price with
discount
 Duplicate Call to Action
at top of page
 Increased conversion
rate
 Increased conversion
rate
 Increased Pageviews on
Checkout Page

12. The goal of the Hypothesis
Gathering insights on your users
A good hypothesis enables you to:
1. Focus on the right goals & KPIs
2. Focus on the right elements
3. Collect data to make data-driven decisions
#OptimizelySchool

13. If______ Then______ Due to______
change result
(quantifiable)
rationale

14. If______ Then______ Due to______
change result
(quantifiable)
rationale
IF we add a size calculator to every product page

15. If______ Then______ Due to______
change result
(quantifiable)
rationale
IF we add a size calculator to every product page THEN the conversion
rate will increase

16. If______ Then______ Due to______
change result
(quantifiable)
rationale
IF we add a size calculator to every product page THEN the conversion
rate will increase DUE TO the increased trust of buying the right size
product.

18. Hypothesis: removing elements will allow for more product copy
and less clutter on the page.
Good hypothesis: if we divide the homepage in a 80-20 ratio
between product related and non product related items
then we will see an increase in article page views,
due to the reduction of distracting elements on the page customers
can better focus on article content.

19. Make sure you learn from every test
A bad hypothesis = No actionable insights

22. Hypothesis:
If we show text suggestive of the range of products we offer
then we can increase the product page views and conversions.
Due to users arriving at their preferred products easier.

23. Hypothesis:
If we integrate weather information from Skymosity into the
predictive text in the search bar
then we can increase the product page views and conversions.
Due to users arriving at their preferred products easier.

25. Hypothesis:
If we hide the coupon code field behind a link
then we can increase the conversions and reduce the drop off rate
due to users not leaving site to go find a coupon code

27. Hypothesis:
If we add a tab to search for vacation packages
then we can increase the average order value
due to the fact that users will engage with a feature they could not
previously considered.

29. Hypothesis:
If we mention all shows on the homepage
then we will increase the overall engagement.
Due to the better visibility of all shows.

30. +600% in Engagement

31. • is easy to find if you follow the Problem, Solution, Result pattern
• Consists of 3 fundamental building block: IF… THEN…DUE TO
• Shows a change in a quantifiable metric
• Gives actionable insights (no matter the outcome)
So a good hypothesis…

32. Annemarie de Jong
Optimization Consultant
 annemarie@optimizely.com
 @annemariedejong

33. Sign Up for the next Class!

34. Until Class 3!