KISSmetrics Summer 2012 State of AB Testing

SUMMER 2012 STATE OF A/B TESTING Lars Lofgren Marketing Analyst - August 2012 info@kissmetrics.com - Confidential - Do not distribute

@larslofgren #KISSwebinar

We’ll cover...1 State of A/B Testing2 Q&A3 Demo

Let’s start with the data!

WHO AREWe?

Who are we? 176 in total 60 45 30 15 0 Ecommerce SaaS Agency/Cons Publishing Mobile Other

What’s the size of our teams? 90 67.5 45 22.5 0 1-5 6-10 11-25 26-60 51-200 200+

What testing tools does everyone use? 90 67.5 45 22.5 0 Visual Website GA Content Optimizely None Other Optimizer Experiments

How many A/B tests did we run last month? 150112.5 75 37.5 0 0 1-4 5-9 10-19 20+

It’s easy to get ahead of our competitors

How do we generate ideas for testing? 60 45 30 15 0 Test Experience Best Practices Research Needs Randomly

Step 1: Get Qualitative

Step 2: Make a prediction

Step 3: Confirm with an A/B test

How KISSmetrics generates testing ideas 1 Get qualitative insights 2 Predict how to improve 3 Confirm prediction with an A/B test

OURResults

Do we regularly increase conversions? 80 60 40 20 0 Always Sometimes Rarely Never

The revenue we gain from A/B testing 70 52.5 35 17.5 0 $1001- $10,001- Over None $1-1,000 No Idea 10,000 100,000 $100,000

2 ways to make money with A/B testing

Option 1: Understand your customers first 1 Get qualitative insights 2 Predict how to improve 3 Confirm prediction with an A/B test

You’ll find the BIG wins much faster

Option 2: Get relentless

How pros like Facebook and Twitter do it 1 Build an entire growth team 2 Back-to-back testing 3 Look for 100s of small improvements

How do we A/B test with small amounts of data?

Do tests with customer interviews 1 Talk to people, give different options 2 Look for trends 3 Build and see if actions match

Do we track results over the long term? 130 97.5 65 32.5 0 Yes No

The pitfalls of not tracking long term results

Variation B performs the best in this funnel

Variation A gives the most revenue

LET’S TRACK THE RETURN ON A/B Testing

First, make sure you can track lifetime value You’ll need customer analytics that can track multiple purchases from the same person.

Where do we find customer analytics?

Second, connect A/B tools to customer analytics OR

What about Google Analytics? Connects tests to goals, not ecommerce transactions.

Q&A Time! Lars Lofgren llofgren@kissmetrics.com @larslofgren

by KISSmetrics on SlideShare on Aug 24, 2012