Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A/B Testing Pitfalls - MeasureCamp London 2015

4,949 views

Published on

A quick presentation about A/B testing pitfalls you should be aware of. Things that influence your A/B tests and the results you get out them. I

Published in: Data & Analytics
  • Be the first to comment

A/B Testing Pitfalls - MeasureCamp London 2015

  1. 1. A/B Testing Pitfalls Michal Parizek (@parezem), Avast #MeasureCamp , March 2015
  2. 2. Low-hanging Fruit (1) 2 business cycles (2) Big enough data sample (minimum of 200 orders per testing experience (3) No bugs in A/B test setup (4) Daily orders/revenue + cumulative orders/revenue (5) Check http://abtestguide.com/calc/ @parezem | #MeasureCamp
  3. 3. Traffic Mix & Seasonality Challenge A/B test results are tight to the traffic and circumstances of a testing period Solution (1) Make sure all testing experiences get the same traffic mix. (2) Avoid special commerce events for A/B testing (Christmas, Black Friday, Valentine’s day etc.). (3) In case you have a seasonal business, A/B test your hypotheses in both on and off season. 3 @parezem | #MeasureCamp
  4. 4. Cross-device A/B Testing Challenge Attribution in cross-device A/B testing. One user, different devices, not the same testing experience Solution (1) Use targeting only to one device type - not solve multiple same 
 device type issue (2) Wait when tools add “user-centric testing” @parezem | #MeasureCamp
  5. 5. Long Purchase Decision Making Process Challenge Customers from your A/B test made the actual decision before your A/B test was launched Solution (1) Target new visitors only. (2) Set micro-conversion goals when an A/B test focuses on an early part of the purchase process First visit on the LP T = 0 T + 28 Purchase!Third visit on the LP T + 12 Your A/B is launched T + 20 Research Finances, laziness, more research, discount coupons @parezem | #MeasureCamp
  6. 6. Research Online, Purchase Offline Challenge Your online A/B tests influence offline purchases too Solution (1) Show discount coupons for offline purchases - each testing experience has unique discount coupon.
 
 Experience A = DSC08A
 Experience B = DSC08B @parezem | #MeasureCamp
  7. 7. Optimising for Maximum CLTV Challenge You don’t want to get more average customers. You do want to get more excellent customers! Solution (1) A survey after a purchase - “Would you recommend us to your friend?” (2) Re-evaluate your A/B tests after few months. (3) Patterns with your historical data - predict CLTV. (4) Immediate insights: e.g. a share of auto-renewal customers buy again recommend stay loyal upgrade @parezem | #MeasureCamp

×