Ux split testing final

4,129 views

Published on

Ux split testing final

  1. 1. UX & Split Testing<br />UX, MVT, A | B testing, UCD, persuasion architecture, web analytics and any other buzz words you can think of...<br />Death by PowerPoint courtesy of Lucy Spence<br />
  2. 2. What are we discussing?<br />Different types of split testing and how those fits into our UX world<br />What it is?<br />Why you should do it?<br /><ul><li>Who should do it?</li></ul>When to use it?<br />How to use it?<br />What to avoid?<br />How does all that fit with UX<br />
  3. 3. What is it? The different types of split tests (terminology)<br />A | B<br />Two versions of the same thing – traffic split 50 | 50<br />Useful for deciding between radically different solutions<br />A | B | n<br />Same as A | B test but with more than 2<br />MVT (multivariate tests)<br />Many different variations of many different section of a page<br />Useful for optimising established funnels<br />
  4. 4. What is it?What sort of things can you test?<br />A | B | n<br />Radically different designs<br />Designs don’t need common elements<br />Templates can be structured in completely distinct ways<br />Content doesn’t need to be shared<br />Focus on:<br />Key messages<br />Content structure<br />MVT<br />Primarily used to test different treatments of elements:<br />Layout tweaks & positioning of elements<br />Text amends<br />Buttons - size, colour, style, labels & icons<br />Removal of particular elements<br />Focus on:<br />Optimizing what is already working for you<br />
  5. 5. A diversion...My first thoughts when I heard about MVT<br />Wow. Cool.<br />OMG! I’m going to be out of a job <br />Hang on – no - wait (thinking, thinking, thinking...) ...maybe there’s a way I can be part of it...<br />Maybe designers no longer design solutions but design experiments to test possible solutions<br />Yay – we can do whatever we want! It’s play time!<br />It doesn’t really matter anyway it’s all justrandom anyway<br />
  6. 6. A diversion... What I’ve discovered (mostly through mistakes)<br />MVT can be fun, informative and deliver strong commercial results<br />MVT can also be painful, confusing, time consuming, divisive and deliver no commercial benefits<br />Its not quite as simple as everyone makes it out to be<br />You need to be familiar with the tools that you’re using and what is / isn’t possible<br />Most only work on a page at a time – yet mostly we’re testing multi step funnels<br />You need to understand statistical significance in order to get results you trust<br />You need to be crystal clear about your goals and what you’re trying to achieve<br />
  7. 7. Why should you do it?What are the benefits?<br />Because on the whole we’re a strange and peculiar species<br />We’re unpredictable - people are strongly influenced by subconscious thoughts and subtle cues in most environments<br />We can’t simulate someone’s actual purchasing decisions<br />We don’t have a better way of measuring the persuasiveness of different solutions<br />Businesses can make quite a bit of money out of it without doing very much<br />
  8. 8. Who should do it?Is it suitable for your organisation?<br />Organisations with lots of traffic - you’ve got to get a reasonable amount of visitors through a test to get a statistically significant result<br />Testing 2 versions of the same page with a 30% conversion rate and 1000 visitors a day will take 77 days to statistically prove a 2% improvement with 95% confidence.<br />If you need to work it out, use a calculator: https://www.google.com/analytics/siteopt/siteopt/help/calculator.html<br />Organisation that are making money through their site<br />It helps clarify your goals<br />Do you have targeted landing pages for PPC activity? If not, it’s probably not for you.<br />Bloggers (this is an exception) - it’s a easy / cheap way to do engagement experiments such as bounce rate or commenting<br />
  9. 9. When to use it?Can I use it for everything I do?<br />It’s not a magic wand - it’s just a tool – use it appropriately<br />Use A | B | n testing early on to help determine direction<br />Use static HTML for tests, don’t bother developing complex solutions<br />Don’t test complex solutions that you’ve already built<br />Use MVT on your final funnel solutions after all other user testing is done.<br />Repeat MVT until you’re struggling to come up with any other options, ie. never stop<br />
  10. 10. How to use it?Where do you start and how do you progress<br />Start off simply – Google’s Website Optimizer is free and easy<br />Build up your level of complexity as you get more experience<br />Different vendors provide different levels of sophistication<br />Factors you might want to consider:<br />Level of support / analysis<br />Level of self service<br />Segmentation <br />Integration with other data centres & web analytics<br />Email, PPC campaigns, mobile, other platforms<br />Move toward the ultimate – full functional split testing / experience testing (but only a very small number will ever make it that far)<br />
  11. 11. What to avoid?What are the pain points?<br />Don’t test something if you’re just going to put it live anyway<br />Multiple sources of lies can build distrust if not reconciled / understood<br />Lift isn’t guaranteed even when testing positively<br />Nothing remains static<br />Seeing trends against natural variation is almost impossible<br />Control limit analysis often doesn’t work – timescales are too long<br />i18n / localization may produce different results<br />
  12. 12. UCD / UX / UED How does it fit with UX?<br />It’s not a magic wand - it’s just a tool – use it appropriately<br />It’s about the only real way to measure persuasiveness accurately<br />Use A | B | n testing early on to help determine direction<br />Use static HTML for tests, don’t bother developing dynamic solutions<br />Use MVT on your final funnel solutions after all other user testing is done.<br />Make what works, work harder<br />The next step – design with MVT in mind<br />Know what elements you want to move around in the funnel<br />
  13. 13. Contact details<br />If you found this interesting and want to stay in touch, or have some questions:<br />Linked in: http://uk.linkedin.com/in/lucyspence<br />Twitter: http://twitter.com/lucyjspence<br />Email: lucyjspence@googlemail.com<br />And if you want a 2 month free trial for LOVEFiLM use the promo code: HBAQRRMDD<br />

×