A/B split testing isn't something you did, it's something you do. To be effective, A/B split testing isn't a one and done exercise. According to Matt Whrel and Matt Beischel, both part of the CohereOne team, A/B testing needs to be part of everyday marketing. How do you test? Surprisingly simple to set up and run, A/B split tests require little more than some strategy, a few snippets of Javascript and access to third party software. You can be up and running for less than you will spend on coffee this month. What should you test? Matt Whrel breaks tests types down into three categories to help you understand the underlying principles of testing, and Matt Beischel provides examples of each along with some interesting results. How often should you test? Both Matts share examples of testing that surprised them when repeated. They offer advice on how to read results and discuss why seasonality can affect A/B test results. What should you test with? Matt Whrel shares the testing tools they use at CohereOne. (Spoiler alert: Crazy Egg and Optimizely.) This Pub Talk is loaded with great information on how to develop a long term, ongoing A/B testing program. It was a lively session with a lot of Q&A at the end.

1. How to build the perfect website with A/B Experiments

2. What is a good conversion rate?
2.2%
3.4%
1.7%
4.5%

3. What is a good conversion rate?
There is no such thing as a good conversion
rate…it is a mythological creature!

4. The purpose of improving conversion…

5. Agenda
• The new world of A/B experiments
• Three different types of experiments
• Experiment examples and results
• The future of A/B experiments

6. Old Objections to A/B split testing
• Setting up experiments will mean making changes to the site.
• Can’t do…I’ve got a six month backlog in website projects
• Website changes are expensive; I don’t have the budget
• Every time I make a change to my website something breaks.
• My ecommerce platform doesn’t allow me to change the things I
want to test.
• My ecommerce platform doesn’t allow me to split test

7. The Solution for A/B experiments
Third-party software
• Use built-in design studio to create changes;
wyswyg or direct html/javascript control
• No changes to the platform code to execute the
test, so doesn’t affect current development or risk
breaking anything on the site.
• Very fast, easy, inexpensive to set-up experiments.

8. Everyday Experiments
• Everyday experiments = continually run and
measure A/B tests on your website.
• Much like your email campaigns, plan and build
experiments in advance and launch them at the
appropriate time – always have an experiment
running.
• Experiments are not a big event; part of your
everyday marketing.

9. Everyday Experiments Results
17%
14%
70%

10. Types of Experiments: 1 – Reduce Friction

11. A lot of Friction

12. A lot easier

13. Friction on the product page

14. Removing Friction
1.
2.
3

15. Product page add to cart results

16. Product Page Order results

17. Product Page Revenue results

18. Types of Experiments: 2 – Add Urgency

19. Urgency – Stock Status Messaging

20. Urgency with stock status messaging

21. Urgency Message – Orders results

22. Urgency with stock status messaging

23. Urgency Message – Orders results

24. Types of Experiments: 3 – Merchandising

25. Merchandising: Category Sort Order

26. Merchandising Category Sort Order

27. Merchandising – Category Sort Order

28. Merchandising – Category Sort Order

29. Short Term Next Frontier -- Personalization

30. Real Next Frontier -- Audiences
Measure the effect of changes on the website at the
audience level and across all transactional channels

31. Questions?
Questions?

32. One Last Thing
Thank You!
Matt Wrhel: mwrhel@cohereone.com
Matt Beischel: mbeischel@cohereone.com