A/B Testing - The Right Way


Published on

A/B Testing... The Right Way: The What's, Why's and How's to do A/B Testing Successfully. Learn how A/B Testing can help you improve your website content and conversion rate, and help you influence the behavior of visitors to your site.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

A/B Testing - The Right Way

  1. 1. A/B Testing…The Right WayThe What’s, Why’s and How’s to do A/B Testing Successfully
  2. 2. Hello!• Holly Wright• Marketing Manager• holly.wright@spectate.com• @hollygowrightly
  3. 3. “I’m still experimenting.” – Stevie Wonder“While you are experimenting, do not remaincontent with the surface of things.” – Ivan Pavlov
  4. 4. A/B Testing• A/B Testing is a method for experiment with your web design and/or content• The purpose of A/B Testing is to improve some aspect of your website, ads, landing pages or other digital content
  5. 5. The Concept• A/B Testing has been used for a long time in marketing and advertising• Traditionally, it has included testing ads in different publications, testing product packing in different markets, testing price points in different geographies
  6. 6. The Basics• The most basic A/B Test for web involves one original piece of content and one variation• You present the two options to visitors, see which one gets the best response and then make that your standard
  7. 7. The Goal• A/B Testing is not about creating the perfect ad, landing page, website, etc.• The goal is just to create a better ad, landing page, website, etc.
  8. 8. The Process• You don’t have to stop after one test.• A/B Testing is all about the iterative process of revising, optimizing and refining
  9. 9. The Experimental Method• Independent Variable• Dependent Variable• Control Variables• Hypothesis
  10. 10. Independent Variables– Images – Delivery Method– Form Fields – Layout– Colors – Length of Form– Fonts – Format of Text– Capitalization – Header Text– Button Size – Required fields– Word Selection – Emails– Pricing Offers– Social Buttons
  11. 11. Dependent Variable• This is the thing you’re testing• It can be just about any behavior: – Clicks – Conversions – Time on Page – Purchase Behavior – Social Shares
  12. 12. The Dependent Variable• You must decide on a behavior to measure in order to “declare a winner”• Measured behavior should match test – In other words, if you’re testing an ad, then you want to measure clicks – If you’re testing a landing page or promotion, you may want to measure conversions or purchases – If you’re testing a content format, you may want to test the time visitors spend on your page
  13. 13. The Experimental Method for a Callout• Independent Variable(s): the callout• Dependent Variable: clicks• Control Variables: logo, font size, images used on page, content on page, layout, colors and menu items (in other words, everything else!)• Hypothesis: A blue callout will get more clicks than a red callout
  14. 14. The Experimental Method for a Design Element• Independent Variable(s): the font size• Dependent Variable: time on page• Control Variables: logo, images used on page, content on page, layout, colors and menu items (in other words, everything else!)• Hypothesis: A larger font size will increase the average time people spend on my homepage
  15. 15. Humans are Irrational
  16. 16. “Engineers think everyone else shouldthink like them. Marketers know thateveryone else doesn’t think like them”
  17. 17. The Ever-Important Hypothesis• You do not have a psychic gift• It’s important to make hypotheses, so you think about what you’re testing and why, but don’t expect to be right much of the time
  18. 18. A/B Testing vs. Multivariate TestingA/B Testing Multivariate Testing• A/B Testing involves two or • Multivariate Testing involves more versions of a single multiple versions of page or page element multiple page elements that• This is a simpler way to test are presented in random visitors’ responsiveness to combinations different messages • This method requires either• It can be relatively quick to more traffic volume or more test this way time in order to “declare a• AKA “split testing” winner”
  19. 19. Which one?• If you don’t have a lot of traffic or a lot of time, the best approach is going to be testing just two versions of a single page or element• If you have more time and/or traffic, you can test three or more versions at a time• If you also have money to invest in a multivariate testing tool, go nuts!
  20. 20. The First Step• If you’re just starting out with this, start broad – Test very noticeable things – Change up the entire design of a page – Use a completely different message
  21. 21. Declaring a Winner• This is all about traffic (which is all about the rate of traffic and time)• You need to have a sufficient number of visitors and the desired behavior before you have enough data to make a decision
  22. 22. Declaring a Winner• If you make a decision too early or with too small a sample size, you could be making your site WORSE!
  23. 23. A Word About Statistical Significance• If your results between the two tests are significantly different, you can make a decision with less data• If your results are very close, you need a lot more data to be confident that one variation is significantly better than the other• The good news is, we don’t have to do the math ourselves…
  24. 24. A Word About Statistical Significance http://getdatadriven.com/ab-significance-test
  25. 25. The Next Steps• If you’ve found a winner from your first test, try refining that version – Test smaller elements to see what converts best – Run a longer test with a few different versions – Get creative--Try tests you’d never expect someone to test, like line spacing, italics, bullet point length – The web is your laboratory!
  26. 26. What If There IS No Winner?• If your test doesn’t yield a clear winner, then keep testing• You can – Extend the time of the test, – Change up one of your samples, or – Just pick one and go with it!
  27. 27. SEO & A/B Testing• Google wants us to do A/B Testing• It’s better for them if our web content is the best it can be!• There are some guidelines for A/B Testing so it won’t affect your SERP placement
  28. 28. SEO & A/B Testing - Guidelines• Don’t do anything sneaky with your JavaScript, like showing different content to bots vs. humans• This is called cloaking and is not a typical A/B Testing practice, but can get you in trouble when you are using JS
  29. 29. SEO & A/B Testing - Guidelines• After a test is over, use 301 (permanent) or 302 (temporary) redirects to your original URL• This ensures that any visitors who have bookmarked your page or linked to it from their sites will always find the right page
  30. 30. SEO & A/B Testing - Guidelines• Use the rel=“conanical” link element if you can’t use redirects• This tells Google which version of a page is your preferred version, which influences the content is serves up in search resultsGoogle’s explanation of this:http://support.google.com/webmasters/bin/answer.py?hl=en&answer=139394
  31. 31. Call-to-Action Example
  32. 32. Call-to-Action Example
  33. 33. Call-to-Action ExampleAB
  34. 34. Call-to-Action Example
  35. 35. Call-to-Action Example
  36. 36. Popular Tools• Google Content Experiments• Optimizely or Visual Website Optimizer• Spectate!
  37. 37. Google Content Experiments• Test up to five different versions of a page and select specific metrics and goals• Essentially A/B Testing for entire pages (or A/B/C/D/E Testing)• Integrates with Google Analytics• FREE!
  38. 38. Optimizely / Visual Website Optimizer• Advanced visual page builders• Allow visitor targeting to customize test• Only need a single line of code to implement• Good reporting features• VWO has more features but a higher base price than Optimizely; pricing is based on monthly visitors
  39. 39. Spectate• Allows A/B Testing of callouts and landing pages• Includes a visual landing page editor• Is not just a testing tool; also includes a host of other features, such as social posting, form builders, email auto-responders, SEO scoring, notification messages, nurture programs, etc.
  40. 40. Putting It All Together• A/B testing can help you optimize your content and improve your conversions• The process is iterative because you can always improve• Search engines have no problem with this as long as you follow some general rules• There are tons of tools out there to assist with both A/B Testing and Multivariate Testing• Statistical significance must be taken into account before deciding on a winner• Dependent variables need to match up with independent variables being tested
  41. 41. Q&A
  42. 42. Resources• http://spectate.com/?s=a%2Fb+testing&submit.x=- 1129&submit.y=-136• http://support.google.com/webmasters/bin/answer.py ?hl=en&answer=139394• http://getdatadriven.com/ab-significance-test• http://rich-page.com/website-optimization/which-low- cost-testing-tool-google-content-experiments-vs- optimizely-vs-visual-website-optimizer/
  43. 43. Contact Info• Holly Wright• Marketing Manager• holly.wright@spectate.com• @hollygowrightly Want to try out Spectate for free? Contact me for a FREE 15-Day Trial!