Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How to Run AB Tests that Get REAL Results

10,456 views

Published on

• 3. We’ll cover...1 The wrong way to A/B test2 5 places to get ideas for testing3 Get the most from your A/B testing tools
• 4. HOW MOST PEOPLEA/B Test
• 5. A/B testing is totally rad! We should do it!
• 6. Everyone loves button colors VS
• 7. Next up? Layouts!
• 8. What about photos? VS
• 9. And conversions went up by 4%! Party time!
• 10. There’s a better way... Testing random elements is very inefficient. You will not find the big wins this way.
• 11. You’re also not learning Red vs green buttons doesn’t help you understand your customer.
• 12. GETQualitative
• 13. Qualitative data definition Data that isn’t a number.
• 14. Start with qualitative insights Use other sources of data to get a basic understanding of what your customers want.
• 15. Understand first, then do the A/B test You’ll focus on the right tests and increase your conversions a LOT faster.
• 16. The A/B Testing Process 1 Get qualitative insights 2 Predict how to improve 3 Confirm prediction with an A/B test
• 17. Why is this process better? Helps you find the 40% increase in conversions instead of only the 4%.
• 18. WHERE IS THE QUALITATIVE Data?
• 19. Location #1 Surveys
• 20. Use one-question surveys on your site
• 21. Also try long surveys
• 22. Survey best practices1 Keep it short2 Only ask for answers that you’ll use3 Start with open-ended questions
• 23. Location #2 Feedback
• 24. Do you have a feedback email?
• 25. Talk to your support team They know exactly what problems keep coming up.
• 26. Also talk to your sales team They talk to customers every day.
• 27. Location #3 Reach Out
• 28. Nothing beats talking to customers personally Get them on the phone and deeply understand how they use your product.
• 29. If local, take them out to lunch or coffee.
• 30. This is not a pitch Focus on asking good questions and listening.
• 31. Location #4 User Activity
• 32. Look at activity of individual people You’ll need customer analytics for this.
• 33. KISSmetrics People report
• 34. When you have an insight... Confirm by looking at other people.
• 35. Location #5Usability Tests
• 36. What’s a usability test? Have someone use your site or product while you watch.
• 37. Grab someone off the street
• 38. Amazon’s Mechanical Turk - mturk.com
• 39. UserTesting.com
• 40. Places to Find Qualitative Insights
• 41. The A/B Testing Process
• 42. When should you test the small stuff? After you’ve found the big wins and you have a full-time growth team.
• 43. A/B TESTINGTools
• 44. A/B testing tools only measure the next step
• 45. Tie A/B tests back to the customer life-cycle
• 46. How do we actually do this?
• 47. First, make sure you’re using customer analytics
• 48. Then use Visual Website Optimizer or Optimizely OR
• 49. Now all your A/B testing data is in KISSmetrics

Published in: Technology, Education
  • Be the first to comment

How to Run AB Tests that Get REAL Results

  1. 1. How to Run A/B Tests that Get REAL Results Lars Lofgren Marketing Analyst - July 2012 info@kissmetrics.com - Confidential - Do not distribute
  2. 2. @larslofgren #KISSwebinar
  3. 3. We’ll cover...1 The wrong way to A/B test2 5 places to get ideas for testing3 Get the most from your A/B testing tools
  4. 4. HOW MOST PEOPLEA/B Test
  5. 5. A/B testing is totally rad! We should do it!
  6. 6. Everyone loves button colors VS
  7. 7. Next up? Layouts!
  8. 8. What about photos? VS
  9. 9. And conversions went up by 4%! Party time!
  10. 10. There’s a better way... Testing random elements is very inefficient. You will not find the big wins this way.
  11. 11. You’re also not learning Red vs green buttons doesn’t help you understand your customer.
  12. 12. GETQualitative
  13. 13. Qualitative data definition Data that isn’t a number.
  14. 14. Start with qualitative insights Use other sources of data to get a basic understanding of what your customers want.
  15. 15. Understand first, then do the A/B test You’ll focus on the right tests and increase your conversions a LOT faster.
  16. 16. The A/B Testing Process 1 Get qualitative insights 2 Predict how to improve 3 Confirm prediction with an A/B test
  17. 17. Why is this process better? Helps you find the 40% increase in conversions instead of only the 4%.
  18. 18. WHERE IS THE QUALITATIVE Data?
  19. 19. Location #1 Surveys
  20. 20. Use one-question surveys on your site
  21. 21. Also try long surveys
  22. 22. Survey best practices1 Keep it short2 Only ask for answers that you’ll use3 Start with open-ended questions
  23. 23. Location #2 Feedback
  24. 24. Do you have a feedback email?
  25. 25. Talk to your support team They know exactly what problems keep coming up.
  26. 26. Also talk to your sales team They talk to customers every day.
  27. 27. Location #3 Reach Out
  28. 28. Nothing beats talking to customers personally Get them on the phone and deeply understand how they use your product.
  29. 29. If local, take them out to lunch or coffee.
  30. 30. This is not a pitch Focus on asking good questions and listening.
  31. 31. Location #4 User Activity
  32. 32. Look at activity of individual people You’ll need customer analytics for this.
  33. 33. KISSmetrics People report
  34. 34. When you have an insight... Confirm by looking at other people.
  35. 35. Location #5Usability Tests
  36. 36. What’s a usability test? Have someone use your site or product while you watch.
  37. 37. Grab someone off the street
  38. 38. Amazon’s Mechanical Turk - mturk.com
  39. 39. UserTesting.com
  40. 40. Places to Find Qualitative Insights 1 Surveys 2 Feedback 3 Reach Out 4 User Activity 5 Usability Tests
  41. 41. The A/B Testing Process 1 Get qualitative insights 2 Predict how to improve 3 Confirm prediction with an A/B test
  42. 42. When should you test the small stuff? After you’ve found the big wins and you have a full-time growth team.
  43. 43. A/B TESTINGTools
  44. 44. A/B testing tools only measure the next step Great for testing copy, photos, and layouts. Terrible for bigger tests like pricing.
  45. 45. Tie A/B tests back to the customer life-cycle Critical for measuring how your changes impact customers over the long term.
  46. 46. How do we actually do this?
  47. 47. First, make sure you’re using customer analytics
  48. 48. Then use Visual Website Optimizer or Optimizely OR
  49. 49. Now all your A/B testing data is in KISSmetrics
  50. 50. For a demo of KISSmetrics or to learnhow KISSmetircs can help grow your business, contact Ben Sardella at: bsardella@kissmetrics.com

×