KISSmetrics Summer 2012 State of AB Testing

2,055 views
1,867 views

Published on

3. We’ll cover...1 State of A/B Testing2 Q&A3 Demo
4. Let’s start with the data!
5. WHO AREWe?
6. Who are we? 176 in total 60 45 30 15 0 Ecommerce SaaS Agency/Cons Publishing Mobile Other
7. What’s the size of our teams? 90 67.5 45 22.5 0 1-5 6-10 11-25 26-60 51-200 200+
8. What testing tools does everyone use? 90 67.5 45 22.5 0 Visual Website GA Content Optimizely None Other Optimizer Experiments
9. How many A/B tests did we run last month? 150112.5 75 37.5 0 0 1-4 5-9 10-19 20+
10. It’s easy to get ahead of our competitors
11. How do we generate ideas for testing? 60 45 30 15 0 Test Experience Best Practices Research Needs Randomly
12. Step 1: Get Qualitative
13. Step 2: Make a prediction
14. Step 3: Confirm with an A/B test
15. How KISSmetrics generates testing ideas 1 Get qualitative insights 2 Predict how to improve 3 Confirm prediction with an A/B test
16. OURResults
17. Do we regularly increase conversions? 80 60 40 20 0 Always Sometimes Rarely Never
18. The revenue we gain from A/B testing 70 52.5 35 17.5 0 $1001- $10,001- Over None $1-1,000 No Idea 10,000 100,000 $100,000
19. 2 ways to make money with A/B testing
20. Option 1: Understand your customers first 1 Get qualitative insights 2 Predict how to improve 3 Confirm prediction with an A/B test
21. You’ll find the BIG wins much faster
22. Option 2: Get relentless
23. How pros like Facebook and Twitter do it 1 Build an entire growth team 2 Back-to-back testing 3 Look for 100s of small improvements
24. How do we A/B test with small amounts of data?
25. Do tests with customer interviews 1 Talk to people, give different options 2 Look for trends 3 Build and see if actions match
26. Do we track results over the long term? 130 97.5 65 32.5 0 Yes No
27. The pitfalls of not tracking long term results
28. Variation B performs the best in this funnel
29. Variation A gives the most revenue
30. LET’S TRACK THE RETURN ON A/B Testing
31. First, make sure you can track lifetime value You’ll need customer analytics that can track multiple purchases from the same person.
32. Where do we find customer analytics?
33. Second, connect A/B tools to customer analytics OR
34. What about Google Analytics? Connects tests to goals, not ecommerce transactions.

Published in: Technology, Education
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,055
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
71
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

KISSmetrics Summer 2012 State of AB Testing

  1. 1. SUMMER 2012 STATE OF A/B TESTING Lars Lofgren Marketing Analyst - August 2012 info@kissmetrics.com - Confidential - Do not distribute
  2. 2. @larslofgren #KISSwebinar
  3. 3. We’ll cover...1 State of A/B Testing2 Q&A3 Demo
  4. 4. Let’s start with the data!
  5. 5. WHO AREWe?
  6. 6. Who are we? 176 in total 60 45 30 15 0 Ecommerce SaaS Agency/Cons Publishing Mobile Other
  7. 7. What’s the size of our teams? 90 67.5 45 22.5 0 1-5 6-10 11-25 26-60 51-200 200+
  8. 8. What testing tools does everyone use? 90 67.5 45 22.5 0 Visual Website GA Content Optimizely None Other Optimizer Experiments
  9. 9. How many A/B tests did we run last month? 150112.5 75 37.5 0 0 1-4 5-9 10-19 20+
  10. 10. It’s easy to get ahead of our competitors
  11. 11. How do we generate ideas for testing? 60 45 30 15 0 Test Experience Best Practices Research Needs Randomly
  12. 12. Step 1: Get Qualitative
  13. 13. Step 2: Make a prediction
  14. 14. Step 3: Confirm with an A/B test
  15. 15. How KISSmetrics generates testing ideas 1 Get qualitative insights 2 Predict how to improve 3 Confirm prediction with an A/B test
  16. 16. OURResults
  17. 17. Do we regularly increase conversions? 80 60 40 20 0 Always Sometimes Rarely Never
  18. 18. The revenue we gain from A/B testing 70 52.5 35 17.5 0 $1001- $10,001- Over None $1-1,000 No Idea 10,000 100,000 $100,000
  19. 19. 2 ways to make money with A/B testing
  20. 20. Option 1: Understand your customers first 1 Get qualitative insights 2 Predict how to improve 3 Confirm prediction with an A/B test
  21. 21. You’ll find the BIG wins much faster
  22. 22. Option 2: Get relentless
  23. 23. How pros like Facebook and Twitter do it 1 Build an entire growth team 2 Back-to-back testing 3 Look for 100s of small improvements
  24. 24. How do we A/B test with small amounts of data?
  25. 25. Do tests with customer interviews 1 Talk to people, give different options 2 Look for trends 3 Build and see if actions match
  26. 26. Do we track results over the long term? 130 97.5 65 32.5 0 Yes No
  27. 27. The pitfalls of not tracking long term results
  28. 28. Variation B performs the best in this funnel
  29. 29. Variation A gives the most revenue
  30. 30. LET’S TRACK THE RETURN ON A/B Testing
  31. 31. First, make sure you can track lifetime value You’ll need customer analytics that can track multiple purchases from the same person.
  32. 32. Where do we find customer analytics?
  33. 33. Second, connect A/B tools to customer analytics OR
  34. 34. What about Google Analytics? Connects tests to goals, not ecommerce transactions.
  35. 35. Q&A Time! Lars Lofgren llofgren@kissmetrics.com @larslofgren

×