How to Run AB Tests that Get REAL Results

9,220 views
9,037 views

Published on

• 3. We’ll cover...1 The wrong way to A/B test2 5 places to get ideas for testing3 Get the most from your A/B testing tools
• 4. HOW MOST PEOPLEA/B Test
• 5. A/B testing is totally rad! We should do it!
• 6. Everyone loves button colors VS
• 7. Next up? Layouts!
• 8. What about photos? VS
• 9. And conversions went up by 4%! Party time!
• 10. There’s a better way... Testing random elements is very inefficient. You will not find the big wins this way.
• 11. You’re also not learning Red vs green buttons doesn’t help you understand your customer.
• 12. GETQualitative
• 13. Qualitative data definition Data that isn’t a number.
• 14. Start with qualitative insights Use other sources of data to get a basic understanding of what your customers want.
• 15. Understand first, then do the A/B test You’ll focus on the right tests and increase your conversions a LOT faster.
• 16. The A/B Testing Process 1 Get qualitative insights 2 Predict how to improve 3 Confirm prediction with an A/B test
• 17. Why is this process better? Helps you find the 40% increase in conversions instead of only the 4%.
• 18. WHERE IS THE QUALITATIVE Data?
• 19. Location #1 Surveys
• 20. Use one-question surveys on your site
• 21. Also try long surveys
• 22. Survey best practices1 Keep it short2 Only ask for answers that you’ll use3 Start with open-ended questions
• 23. Location #2 Feedback
• 24. Do you have a feedback email?
• 25. Talk to your support team They know exactly what problems keep coming up.
• 26. Also talk to your sales team They talk to customers every day.
• 27. Location #3 Reach Out
• 28. Nothing beats talking to customers personally Get them on the phone and deeply understand how they use your product.
• 29. If local, take them out to lunch or coffee.
• 30. This is not a pitch Focus on asking good questions and listening.
• 31. Location #4 User Activity
• 32. Look at activity of individual people You’ll need customer analytics for this.
• 33. KISSmetrics People report
• 34. When you have an insight... Confirm by looking at other people.
• 35. Location #5Usability Tests
• 36. What’s a usability test? Have someone use your site or product while you watch.
• 37. Grab someone off the street
• 38. Amazon’s Mechanical Turk - mturk.com
• 39. UserTesting.com
• 40. Places to Find Qualitative Insights
• 41. The A/B Testing Process
• 42. When should you test the small stuff? After you’ve found the big wins and you have a full-time growth team.
• 43. A/B TESTINGTools
• 44. A/B testing tools only measure the next step
• 45. Tie A/B tests back to the customer life-cycle
• 46. How do we actually do this?
• 47. First, make sure you’re using customer analytics
• 48. Then use Visual Website Optimizer or Optimizely OR
• 49. Now all your A/B testing data is in KISSmetrics

Published in: Technology, Education
0 Comments
36 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
9,220
On SlideShare
0
From Embeds
0
Number of Embeds
2,067
Actions
Shares
0
Downloads
218
Comments
0
Likes
36
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • Let’s take an Apple product for example.\nIt’s obvious to use Apple as a case study for user experience. Their whole business model is built on UX and delighting their customers.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Let’s take an Apple product for example.\nIt’s obvious to use Apple as a case study for user experience. Their whole business model is built on UX and delighting their customers.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Let’s take an Apple product for example.\nIt’s obvious to use Apple as a case study for user experience. Their whole business model is built on UX and delighting their customers.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Let’s take an Apple product for example.\nIt’s obvious to use Apple as a case study for user experience. Their whole business model is built on UX and delighting their customers.\n
  • Let’s take an Apple product for example.\nIt’s obvious to use Apple as a case study for user experience. Their whole business model is built on UX and delighting their customers.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • How to Run AB Tests that Get REAL Results

    1. 1. How to Run A/B Tests that Get REAL Results Lars Lofgren Marketing Analyst - July 2012 info@kissmetrics.com - Confidential - Do not distribute
    2. 2. @larslofgren #KISSwebinar
    3. 3. We’ll cover...1 The wrong way to A/B test2 5 places to get ideas for testing3 Get the most from your A/B testing tools
    4. 4. HOW MOST PEOPLEA/B Test
    5. 5. A/B testing is totally rad! We should do it!
    6. 6. Everyone loves button colors VS
    7. 7. Next up? Layouts!
    8. 8. What about photos? VS
    9. 9. And conversions went up by 4%! Party time!
    10. 10. There’s a better way... Testing random elements is very inefficient. You will not find the big wins this way.
    11. 11. You’re also not learning Red vs green buttons doesn’t help you understand your customer.
    12. 12. GETQualitative
    13. 13. Qualitative data definition Data that isn’t a number.
    14. 14. Start with qualitative insights Use other sources of data to get a basic understanding of what your customers want.
    15. 15. Understand first, then do the A/B test You’ll focus on the right tests and increase your conversions a LOT faster.
    16. 16. The A/B Testing Process 1 Get qualitative insights 2 Predict how to improve 3 Confirm prediction with an A/B test
    17. 17. Why is this process better? Helps you find the 40% increase in conversions instead of only the 4%.
    18. 18. WHERE IS THE QUALITATIVE Data?
    19. 19. Location #1 Surveys
    20. 20. Use one-question surveys on your site
    21. 21. Also try long surveys
    22. 22. Survey best practices1 Keep it short2 Only ask for answers that you’ll use3 Start with open-ended questions
    23. 23. Location #2 Feedback
    24. 24. Do you have a feedback email?
    25. 25. Talk to your support team They know exactly what problems keep coming up.
    26. 26. Also talk to your sales team They talk to customers every day.
    27. 27. Location #3 Reach Out
    28. 28. Nothing beats talking to customers personally Get them on the phone and deeply understand how they use your product.
    29. 29. If local, take them out to lunch or coffee.
    30. 30. This is not a pitch Focus on asking good questions and listening.
    31. 31. Location #4 User Activity
    32. 32. Look at activity of individual people You’ll need customer analytics for this.
    33. 33. KISSmetrics People report
    34. 34. When you have an insight... Confirm by looking at other people.
    35. 35. Location #5Usability Tests
    36. 36. What’s a usability test? Have someone use your site or product while you watch.
    37. 37. Grab someone off the street
    38. 38. Amazon’s Mechanical Turk - mturk.com
    39. 39. UserTesting.com
    40. 40. Places to Find Qualitative Insights 1 Surveys 2 Feedback 3 Reach Out 4 User Activity 5 Usability Tests
    41. 41. The A/B Testing Process 1 Get qualitative insights 2 Predict how to improve 3 Confirm prediction with an A/B test
    42. 42. When should you test the small stuff? After you’ve found the big wins and you have a full-time growth team.
    43. 43. A/B TESTINGTools
    44. 44. A/B testing tools only measure the next step Great for testing copy, photos, and layouts. Terrible for bigger tests like pricing.
    45. 45. Tie A/B tests back to the customer life-cycle Critical for measuring how your changes impact customers over the long term.
    46. 46. How do we actually do this?
    47. 47. First, make sure you’re using customer analytics
    48. 48. Then use Visual Website Optimizer or Optimizely OR
    49. 49. Now all your A/B testing data is in KISSmetrics
    50. 50. For a demo of KISSmetrics or to learnhow KISSmetircs can help grow your business, contact Ben Sardella at: bsardella@kissmetrics.com

    ×