• 3. We’ll cover...1 The wrong way to A/B test2 5 places to get ideas for testing3 Get the most from your A/B testing tools • 4. HOW MOST PEOPLEA/B Test • 5. A/B testing is totally rad! We should do it! • 6. Everyone loves button colors VS • 7. Next up? Layouts! • 8. What about photos? VS • 9. And conversions went up by 4%! Party time! • 10. There’s a better way... Testing random elements is very inefficient. You will not find the big wins this way. • 11. You’re also not learning Red vs green buttons doesn’t help you understand your customer. • 12. GETQualitative • 13. Qualitative data definition Data that isn’t a number. • 14. Start with qualitative insights Use other sources of data to get a basic understanding of what your customers want. • 15. Understand first, then do the A/B test You’ll focus on the right tests and increase your conversions a LOT faster. • 16. The A/B Testing Process 1 Get qualitative insights 2 Predict how to improve 3 Confirm prediction with an A/B test • 17. Why is this process better? Helps you find the 40% increase in conversions instead of only the 4%. • 18. WHERE IS THE QUALITATIVE Data? • 19. Location #1 Surveys • 20. Use one-question surveys on your site • 21. Also try long surveys • 22. Survey best practices1 Keep it short2 Only ask for answers that you’ll use3 Start with open-ended questions • 23. Location #2 Feedback • 24. Do you have a feedback email? • 25. Talk to your support team They know exactly what problems keep coming up. • 26. Also talk to your sales team They talk to customers every day. • 27. Location #3 Reach Out • 28. Nothing beats talking to customers personally Get them on the phone and deeply understand how they use your product. • 29. If local, take them out to lunch or coffee. • 30. This is not a pitch Focus on asking good questions and listening. • 31. Location #4 User Activity • 32. Look at activity of individual people You’ll need customer analytics for this. • 33. KISSmetrics People report • 34. When you have an insight... Confirm by looking at other people. • 35. Location #5Usability Tests • 36. What’s a usability test? Have someone use your site or product while you watch. • 37. Grab someone off the street • 38. Amazon’s Mechanical Turk - mturk.com • 39. UserTesting.com • 40. Places to Find Qualitative Insights • 41. The A/B Testing Process • 42. When should you test the small stuff? After you’ve found the big wins and you have a full-time growth team. • 43. A/B TESTINGTools • 44. A/B testing tools only measure the next step • 45. Tie A/B tests back to the customer life-cycle • 46. How do we actually do this? • 47. First, make sure you’re using customer analytics • 48. Then use Visual Website Optimizer or Optimizely OR • 49. Now all your A/B testing data is in KISSmetrics

1. How to Run A/B Tests that Get REAL Results
Lars Lofgren
Marketing Analyst - July 2012 info@kissmetrics.com - Confidential - Do not distribute

2. @larslofgren #KISSwebinar

3. We’ll cover...
1 The wrong way to A/B test
2 5 places to get ideas for testing
3 Get the most from your A/B testing tools

4. HOW MOST PEOPLE
A/B Test

5. A/B testing is totally rad! We should do it!

6. Everyone loves button colors
VS

7. Next up? Layouts!

8. What about photos?
VS

9. And conversions went up by 4%! Party time!

10. There’s a better way...
Testing random elements is very
inefficient. You will not find the big
wins this way.

11. You’re also not learning
Red vs green buttons doesn’t help you
understand your customer.

12. GET
Qualitative

13. Qualitative data definition
Data that isn’t a number.

14. Start with qualitative insights
Use other sources of data to get a
basic understanding of what your
customers want.

15. Understand first, then do the A/B test
You’ll focus on the right tests and
increase your conversions a LOT
faster.

16. The A/B Testing Process
1 Get qualitative insights
2 Predict how to improve
3 Confirm prediction with an A/B test

17. Why is this process better?
Helps you find the 40% increase in
conversions instead of only the 4%.

18. WHERE IS THE QUALITATIVE
Data?

19. Location #1
Surveys

20. Use one-question surveys on your site

21. Also try long surveys

22. Survey best practices
1 Keep it short
2 Only ask for answers that you’ll use
3 Start with open-ended questions

23. Location #2
Feedback

24. Do you have a feedback email?

25. Talk to your support team
They know exactly what problems
keep coming up.

26. Also talk to your sales team
They talk to customers every day.

27. Location #3
Reach Out

28. Nothing beats talking to customers personally
Get them on the phone and deeply
understand how they use your
product.

29. If local, take them out to lunch or coffee.

30. This is not a pitch
Focus on asking good questions and
listening.

31. Location #4
User Activity

32. Look at activity of individual people
You’ll need customer analytics for
this.

33. KISSmetrics People report

34. When you have an insight...
Confirm by looking at other people.

35. Location #5
Usability Tests

36. What’s a usability test?
Have someone use your site or
product while you watch.

37. Grab someone off the street

38. Amazon’s Mechanical Turk - mturk.com

39. UserTesting.com

40. Places to Find Qualitative Insights
1 Surveys
2 Feedback
3 Reach Out
4 User Activity
5 Usability Tests

41. The A/B Testing Process
1 Get qualitative insights
2 Predict how to improve
3 Confirm prediction with an A/B test

42. When should you test the small stuff?
After you’ve found the big wins and
you have a full-time growth team.

43. A/B TESTING
Tools

44. A/B testing tools only measure the next step
Great for testing copy, photos, and
layouts. Terrible for bigger tests like
pricing.

45. Tie A/B tests back to the customer life-cycle
Critical for measuring how your
changes impact customers over the
long term.

46. How do we actually do this?

47. First, make sure you’re using customer analytics

48. Then use Visual Website Optimizer or Optimizely
OR

49. Now all your A/B testing data is in KISSmetrics

50. For a demo of KISSmetrics or to learn
how KISSmetircs can help grow your
business, contact Ben Sardella at:
bsardella@kissmetrics.com