Testing & Optimisation - Irene Kalkanis


Published on

This presentation was given at Mezzo Labs' Getting Ahead in Web Analytics event in February 2014. It covers at a high level, the tools and techniques used in A/B testing, multivariate testing (MVT) and optimisation.

Published in: Business
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Work intensive
  • This can be explicit – meaning the user must use a profileOr it can be implicit – meaning the segment is applied based on their behaviour
  • Testing & Optimisation - Irene Kalkanis

    1. 1. Testing & Optimisation What is it and how can it help your business? Irene Kalkanis, February 2014
    2. 2. We’ll find out: • What ‘testing and optimisation’ means • How to figure out whether you need it • How to choose the right tool • How to make sure you get value
    3. 3. What is Optimisation Testing? Increase conversions from landing pages through to sale or goal completion Decrease bounce rate from your landing page Make your content / services / products easier to find and easier to buy NOT Search Engine Optimisation (SEO) NOT a substitute for web analytics
    4. 4. What is an A/B test? Buy Buy 50% conversion 20% conversion
    5. 5. What is an A/B/n test? 50% conversion 20% conversion 80% conversion 60% conversion Copy ------- ------- ------- ------- ------- ----- Buy Copy ------------ ------------------- ------------------- Add to Baske t … you can add as many variations as you like, each one needs to be designed separately Read more Bu y
    6. 6. What is an MVT test? Buy Buy 50% conversion 20% conversion 20% conversion 70% conversion Add to Basket Save Buy now Buy now … add more versions of each element and the tool will generate many different combinations of these elements
    7. 7. A/B/n Testing MVT Testing No real constrictions on which creative or page design can be tested Test many small page elements and measure their impact on conversion rate Test dramatic lay out and design differences Measure effect of interaction between elements Less traffic required to reach clear statistical significance Make small, incremental changes to your site to reduce the ‘yuck’ effect Ability to analyse effects of different elements and combinations in detail A/B/n Testing MVT Testing Tests need to be planned carefully to ensure that we understand why one version wins over another Requires much more traffic to reach statistical significance All test pages must be designed manually Usually give approximate results Can lead to sudden, major changes which can result in the ‘yuck’ effect The methodology of focusing on small elements can constrain creativity Can’t test major layout changes easily Pros Cons
    8. 8. Do I need to test my site? The first step is admitting you have a problem
    9. 9. Greenpeace website 2004 Can you spot the call to action?
    10. 10. Greenpeace website 2014 DONATE!! LIKE!! PROTECT TIGERS!!!!
    11. 11. Every website can be optimised You’re already optimising by trial and error, a testing tool will let you: • make decisions informed by what your audience actually does rather than what they tell you, or your own ‘gut feeling’ • learn what works much faster • test changes on small portions of your audience before going live • test your changes and hypotheses in a scientific way
    12. 12. These tools offer both MVT and A/B/n testing Some tools are more focused on UX Some tools are more focused on marketing Many will design variants, set up the tests and analyse the results for you Different tools & set ups work better for different sites, vendors will be able to tell you what they can do for your site Remember that you don’t have to use the same vendor for different tools
    13. 13. Make sure you get value from your tool • Invest in a good support structure - It takes a village: optimisation testing can be work intensive and needs buy in from stakeholders up front - You may need creative and analysts to build and understand your tests - Heat mapping tools help you dig deeper into A/B/n tests • Don’t test everything - Agree on specific hypotheses you want to test - Agree exactly what a ‘conversion’ means to your business - Focus your tests on landing pages and key conversion journeys • Don’t test forever - If you haven’t reached a statistically significant result within a pre-agreed period of time there’s no reason to make a change
    14. 14. Crazy egg heatmap – confetti view
    15. 15. Expedia drops optional form, earns $12m profit Two optimisation case studies from e-consultancy.com ASOS halves cart abandonment by losing the word ‘register’
    16. 16. These tools also offer targeting and ‘personalisation’
    17. 17. Summary 1. A/B and MVT testing: a way to optimise you site by test different versions at the same time 2. Picking the right tool: get information and quotes from vendors, seek non-partial advice from an expert 3. Get value: use tools wisely on well known problems, build a good team of creative and analysts around the tool