This presentation was given at Mezzo Labs' Getting Ahead in Web Analytics event in February 2014. It covers at a high level, the tools and techniques used in A/B testing, multivariate testing (MVT) and optimisation.

1. Testing & Optimisation
What is it and how can it help your business?
Irene Kalkanis, February 2014

2. We’ll find out:
• What ‘testing and optimisation’ means
• How to figure out whether you need it
• How to choose the right tool
• How to make sure you get value

3. What is Optimisation Testing?
Increase conversions from landing pages through to sale or goal
completion
Decrease bounce rate from your landing page
Make your content / services / products easier to find and easier to
buy
NOT Search Engine Optimisation (SEO)
NOT a substitute for web analytics

4. What is an A/B test?
Buy Buy
50% conversion 20% conversion

5. What is an A/B/n test?
50% conversion 20% conversion 80% conversion 60% conversion
Copy
-------
-------
-------
-------
-------
----- Buy
Copy ------------
-------------------
-------------------
Add
to
Baske
t
… you can add as
many variations
as you like, each
one needs to be
designed
separately
Read more
Bu
y

6. What is an MVT test?
Buy Buy
50% conversion 20% conversion 20% conversion 70% conversion
Add to
Basket
Save Buy now
Buy
now
… add more
versions of each
element and the
tool will generate
many different
combinations of
these elements

7. A/B/n Testing MVT Testing
No real constrictions on which creative or page design
can be tested
Test many small page elements and measure their
impact on conversion rate
Test dramatic lay out and design differences Measure effect of interaction between elements
Less traffic required to reach clear statistical
significance
Make small, incremental changes to your site to reduce
the ‘yuck’ effect
Ability to analyse effects of different elements and
combinations in detail
A/B/n Testing MVT Testing
Tests need to be planned carefully to ensure that we
understand why one version wins over another
Requires much more traffic to reach statistical
significance
All test pages must be designed manually Usually give approximate results
Can lead to sudden, major changes which can result in
the ‘yuck’ effect
The methodology of focusing on small elements can
constrain creativity
Can’t test major layout changes easily
Pros
Cons

8. Do I need to test my site?
The first step is admitting you have a
problem

9. Greenpeace website 2004
Can you spot the call to action?

10. Greenpeace website 2014
DONATE!! LIKE!! PROTECT TIGERS!!!!

11. Every website can be optimised
You’re already optimising by trial and error, a testing tool will let
you:
• make decisions informed by what your audience actually does rather
than what they tell you, or your own ‘gut feeling’
• learn what works much faster
• test changes on small portions of your audience before going live
• test your changes and hypotheses in a scientific way

12. These tools offer both MVT
and A/B/n testing
Some tools are more
focused on UX
Some tools are more
focused on marketing
Many will design
variants, set up
the tests and
analyse the
results for you
Different tools &
set ups work
better for
different sites,
vendors will be
able to tell you
what they can do
for your site
Remember that you
don’t have to use the
same vendor for
different tools

13. Make sure you get value from your tool
• Invest in a good support structure
- It takes a village: optimisation testing can be work intensive and needs buy in from stakeholders
up front
- You may need creative and analysts to build and understand your tests
- Heat mapping tools help you dig deeper into A/B/n tests
• Don’t test everything
- Agree on specific hypotheses you want to test
- Agree exactly what a ‘conversion’ means to your business
- Focus your tests on landing pages and key conversion journeys
• Don’t test forever
- If you haven’t reached a statistically significant result within a pre-agreed period of time there’s
no reason to make a change

14. Crazy egg heatmap – confetti view

15. Expedia drops optional form, earns $12m profit
Two optimisation case studies
from e-consultancy.com
ASOS halves cart abandonment by losing the word
‘register’

16. These tools also offer
targeting and
‘personalisation’

17. Summary
1. A/B and MVT testing: a way to optimise you site by test different versions at the same
time
2. Picking the right tool: get information and quotes from vendors, seek non-partial advice
from an expert
3. Get value: use tools wisely on well known problems, build a good team of creative and
analysts around the tool