KISSmetrics Summer 2012 State of AB Testing
 

KISSmetrics Summer 2012 State of AB Testing

on

  • 2,031 views

3. We’ll cover...1 State of A/B Testing2 Q&A3 Demo ...

3. We’ll cover...1 State of A/B Testing2 Q&A3 Demo
4. Let’s start with the data!
5. WHO AREWe?
6. Who are we? 176 in total 60 45 30 15 0 Ecommerce SaaS Agency/Cons Publishing Mobile Other
7. What’s the size of our teams? 90 67.5 45 22.5 0 1-5 6-10 11-25 26-60 51-200 200+
8. What testing tools does everyone use? 90 67.5 45 22.5 0 Visual Website GA Content Optimizely None Other Optimizer Experiments
9. How many A/B tests did we run last month? 150112.5 75 37.5 0 0 1-4 5-9 10-19 20+
10. It’s easy to get ahead of our competitors
11. How do we generate ideas for testing? 60 45 30 15 0 Test Experience Best Practices Research Needs Randomly
12. Step 1: Get Qualitative
13. Step 2: Make a prediction
14. Step 3: Confirm with an A/B test
15. How KISSmetrics generates testing ideas 1 Get qualitative insights 2 Predict how to improve 3 Confirm prediction with an A/B test
16. OURResults
17. Do we regularly increase conversions? 80 60 40 20 0 Always Sometimes Rarely Never
18. The revenue we gain from A/B testing 70 52.5 35 17.5 0 $1001- $10,001- Over None $1-1,000 No Idea 10,000 100,000 $100,000
19. 2 ways to make money with A/B testing
20. Option 1: Understand your customers first 1 Get qualitative insights 2 Predict how to improve 3 Confirm prediction with an A/B test
21. You’ll find the BIG wins much faster
22. Option 2: Get relentless
23. How pros like Facebook and Twitter do it 1 Build an entire growth team 2 Back-to-back testing 3 Look for 100s of small improvements
24. How do we A/B test with small amounts of data?
25. Do tests with customer interviews 1 Talk to people, give different options 2 Look for trends 3 Build and see if actions match
26. Do we track results over the long term? 130 97.5 65 32.5 0 Yes No
27. The pitfalls of not tracking long term results
28. Variation B performs the best in this funnel
29. Variation A gives the most revenue
30. LET’S TRACK THE RETURN ON A/B Testing
31. First, make sure you can track lifetime value You’ll need customer analytics that can track multiple purchases from the same person.
32. Where do we find customer analytics?
33. Second, connect A/B tools to customer analytics OR
34. What about Google Analytics? Connects tests to goals, not ecommerce transactions.

Statistics

Views

Total Views
2,031
Views on SlideShare
2,029
Embed Views
2

Actions

Likes
3
Downloads
69
Comments
0

1 Embed 2

https://si0.twimg.com 2

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

KISSmetrics Summer 2012 State of AB Testing KISSmetrics Summer 2012 State of AB Testing Presentation Transcript

  • SUMMER 2012 STATE OF A/B TESTING Lars Lofgren Marketing Analyst - August 2012 info@kissmetrics.com - Confidential - Do not distribute
  • @larslofgren #KISSwebinar
  • We’ll cover...1 State of A/B Testing2 Q&A3 Demo
  • Let’s start with the data!
  • WHO AREWe?
  • Who are we? 176 in total 60 45 30 15 0 Ecommerce SaaS Agency/Cons Publishing Mobile Other
  • What’s the size of our teams? 90 67.5 45 22.5 0 1-5 6-10 11-25 26-60 51-200 200+
  • What testing tools does everyone use? 90 67.5 45 22.5 0 Visual Website GA Content Optimizely None Other Optimizer Experiments
  • How many A/B tests did we run last month? 150112.5 75 37.5 0 0 1-4 5-9 10-19 20+
  • It’s easy to get ahead of our competitors
  • How do we generate ideas for testing? 60 45 30 15 0 Test Experience Best Practices Research Needs Randomly
  • Step 1: Get Qualitative
  • Step 2: Make a prediction
  • Step 3: Confirm with an A/B test
  • How KISSmetrics generates testing ideas 1 Get qualitative insights 2 Predict how to improve 3 Confirm prediction with an A/B test
  • OURResults
  • Do we regularly increase conversions? 80 60 40 20 0 Always Sometimes Rarely Never
  • The revenue we gain from A/B testing 70 52.5 35 17.5 0 $1001- $10,001- Over None $1-1,000 No Idea 10,000 100,000 $100,000
  • 2 ways to make money with A/B testing
  • Option 1: Understand your customers first 1 Get qualitative insights 2 Predict how to improve 3 Confirm prediction with an A/B test
  • You’ll find the BIG wins much faster
  • Option 2: Get relentless
  • How pros like Facebook and Twitter do it 1 Build an entire growth team 2 Back-to-back testing 3 Look for 100s of small improvements
  • How do we A/B test with small amounts of data?
  • Do tests with customer interviews 1 Talk to people, give different options 2 Look for trends 3 Build and see if actions match
  • Do we track results over the long term? 130 97.5 65 32.5 0 Yes No
  • The pitfalls of not tracking long term results
  • Variation B performs the best in this funnel
  • Variation A gives the most revenue
  • LET’S TRACK THE RETURN ON A/B Testing
  • First, make sure you can track lifetime value You’ll need customer analytics that can track multiple purchases from the same person.
  • Where do we find customer analytics?
  • Second, connect A/B tools to customer analytics OR
  • What about Google Analytics? Connects tests to goals, not ecommerce transactions.
  • Q&A Time! Lars Lofgren llofgren@kissmetrics.com @larslofgren