Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Keynote Digital Data Tips Tuesday - Amsterdam - June 24th 2014 - A/B-testing analytics errors

11,584 views

Published on

Keynote by @tonw / Ton Wesseling - CEO of http://testing.agency - he shares all the failures he saw and also made himself on analytics and A/B-testing.

Are you also questioning why your total website revenue is not exploding after you've put all those A/B-test winners live? Find the answers in these slides.

Published in: Marketing, Technology, Business

Keynote Digital Data Tips Tuesday - Amsterdam - June 24th 2014 - A/B-testing analytics errors

  1. 1. < H a v e m o r e f u n A / B - t e s t i n g > A/B-t e s t i n g Errors To n We s s e l i n g – C EO Te s t i n g . A g e n c y #DDTT A m s t e r d a m 2 0 1 4 / 0 6 / 2 4
  2. 2. Why me? ü 10 years+ A/B-test experience ü I always loved the numbers side ü I’ve made all these mistakes Email: ton@testing.agency Twitter: @TonW #DDTT
  3. 3. Email: ton@testing.agency Twitter: @TonW #DDTT
  4. 4. We optimize Email: ton@testing.agency Twitter: @TonW #DDTT
  5. 5. Nature Email: ton@testing.agency Twitter: @TonW #DDTT
  6. 6. A better example Email: ton@testing.agency Twitter: @TonW #DDTT
  7. 7. Groundhog Day – The A/B-test movie Email: ton@testing.agency Twitter: @TonW #DDTT
  8. 8. Trying to get the girl Email: ton@testing.agency Twitter: @TonW #DDTT
  9. 9. Constant testing environment A / B Testing Email: ton@testing.agency Twitter: @TonW #DDTT
  10. 10. Better Dialogue Optimization Default Variation B Variation Winner!" A Email: ton@testing.agency Twitter: @TonW #DDTT
  11. 11. This is why you’re A/B-testing Small steps to learn how to make the next big step Email: ton@testing.agency Twitter: @TonW #DDTT
  12. 12. Testing is not free Use it to LEARN Email: ton@testing.agency Twitter: @TonW #DDTT
  13. 13. 50% of the A/B-tests I get to see is wrong Email: ton@testing.agency Twitter: @TonW #DDTT
  14. 14. Look at your A/B-testing results! +24% +52% +126% +83% +12% Email: ton@testing.agency Twitter: @TonW #DDTT +16%
  15. 15. Did you push the winners live to your site? You must be rich now! Email: ton@testing.agency Twitter: @TonW #DDTT
  16. 16. Or is this reality? Email: ton@testing.agency Twitter: @TonW #DDTT
  17. 17. Or even worse: A/B-test winners are letting you down! Email: ton@testing.agency Twitter: @TonW #DDTT
  18. 18. Are these real winners? +24% +52% +83% +12% +126% Email: ton@testing.agency Twitter: @TonW #DDTT +16%
  19. 19. 20 reasons why Your A/B-test analyzing sucks Email: ton@testing.agency Twitter: @TonW #DDTT
  20. 20. Get your statistics right! Email: ton@testing.agency Twitter: @TonW #DDTT
  21. 21. 1: low significance levels ü Big chance the winner is not a real winner Email: ton@testing.agency Twitter: @TonW #DDTT
  22. 22. 2: low power levels ü Big chance a real winner is not recognized Email: ton@testing.agency Twitter: @TonW #DDTT
  23. 23. Example test set-up ü 90% significance level ü 100 conversions per variation ü Average conversion rate: 2% ü 20 out of 100 ideas is a winner ü With an average uplift of 10% Email: ton@testing.agency Twitter: @TonW #DDTT
  24. 24. Average: 75% significance & 40% power Email: ton@testing.agency Twitter: @TonW #DDTT
  25. 25. So: ü 100 tests will give 10 false positives (90% significance) ü 40% of the 20 winners are recognized (40% power) ü 8 real winners & 10 false positives ü NOT 18 improvements of 15%+ Email: ton@testing.agency Twitter: @TonW #DDTT
  26. 26. 3: Not calculating traffic up front ü You thought the page had thousands of visitors Email: ton@testing.agency Twitter: @TonW #DDTT
  27. 27. 4: Not calculating what you can improve ü You thought the page had lots and lots conversions Email: ton@testing.agency Twitter: @TonW #DDTT
  28. 28. ABTestGuide.com/calc Email: ton@testing.agency Twitter: @TonW #DDTT
  29. 29. 5: Not choosing your segments up front ü You just go and dig unitil you find something (you will always find something that seems true BUT is not!) Email: ton@testing.agency Twitter: @TonW #DDTT
  30. 30. 6: You stop the test once it is significant Email: ton@testing.agency Twitter: @TonW #DDTT
  31. 31. 6: You stop the test once it is significant Email: ton@testing.agency Twitter: @TonW #DDTT
  32. 32. 7: You just let it run until it’s significant Email: ton@testing.agency Twitter: @TonW #DDTT
  33. 33. 7: You just let it run until it’s significant Your test should have a fixed lenght! Email: ton@testing.agency Twitter: @TonW #DDTT
  34. 34. 8: Your test takes too long ü Cookie deletion (you will loose 10% in two weeks) They will re-enter the the and pollute your samples! Email: ton@testing.agency Twitter: @TonW #DDTT
  35. 35. 9: You just choose a test length ü At least 1 full purchase cycle! Email: ton@testing.agency Twitter: @TonW #DDTT
  36. 36. 10: You don’t test full weeks ü At least 1 full week (if your not testing for time of day / week effects) Email: ton@testing.agency Twitter: @TonW #DDTT
  37. 37. 11: You think they always use one decive ü Only test 1 device at a time! Email: ton@testing.agency Twitter: @TonW #DDTT
  38. 38. 12: You’re testing broken stuff ü Always check browser compatibility! ü Dont’ break dynamic stuff! Email: ton@testing.agency Twitter: @TonW #DDTT
  39. 39. 13: You have multiple tests that overlap ü Your variation did not won (it was a combination of variations) Email: ton@testing.agency Twitter: @TonW #DDTT
  40. 40. 14: You thought this page matters ü But it did not (so you decide to use some other metric to declare the winner) Email: ton@testing.agency Twitter: @TonW #DDTT
  41. 41. 15: You forgot to leave out some visitors ü Only test on those who can be changed (if you test a sales promotion – leave out your current clients who just logon) Email: ton@testing.agency Twitter: @TonW #DDTT
  42. 42. 16: You’re not slowing the control down ü Add the same code to the control Email: ton@testing.agency Twitter: @TonW #DDTT
  43. 43. 17: You’re not testing on fresh people ü Start with control for a full purchase cycle (and then start sending traffic to your variations – setting the control to 0%) Email: ton@testing.agency Twitter: @TonW #DDTT
  44. 44. 18: You forgot to give them time to buy ü Stop getting fresh visitors after your test time (send new traffic to the control, but give you tested people time to finish converting) Email: ton@testing.agency Twitter: @TonW #DDTT
  45. 45. 19: You’re not diving into your analytics ü Always measure with analytics software (You will find conversions that are not supposed to be there!) Email: ton@testing.agency Twitter: @TonW #DDTT
  46. 46. 20: You’re not measuring the right stuff ü Use unique visitors ü Against lifetime value predictors! Email: ton@testing.agency Twitter: @TonW #DDTT
  47. 47. If you keep on making mistakes: Stuff like this happens Email: ton@testing.agency Twitter: @TonW #DDTT
  48. 48. Testing is not free Use it to LEARN Email: ton@testing.agency Twitter: @TonW #DDTT
  49. 49. Bonus: You did not re-test your winners ü Always re-test to be sure it was not a glitch Email: ton@testing.agency Twitter: @TonW #DDTT
  50. 50. So when you implement winners This will happen! Email: ton@testing.agency Twitter: @TonW #DDTT
  51. 51. < H a v e m o r e f u n A / B - t e s t i n g > A/B-t e s t i n g Errors To n We s s e l i n g – Te s t Wi n n e r #DDTT A m s t e r d a m 2 0 1 4 / 0 6 / 2 4
  52. 52. ABTestGuide.com/calc Email: ton@testing.agency Twitter: @TonW #DDTT

×