Keynote Digital Data Tips Tuesday - Amsterdam - June 24th 2014 - A/B-testing analytics errors

10,029 views
9,960 views

Published on

Keynote by @tonw / Ton Wesseling - CEO of http://testing.agency - he shares all the failures he saw and also made himself on analytics and A/B-testing.

Are you also questioning why your total website revenue is not exploding after you've put all those A/B-test winners live? Find the answers in these slides.

Published in: Marketing, Technology, Business
1 Comment
16 Likes
Statistics
Notes
No Downloads
Views
Total views
10,029
On SlideShare
0
From Embeds
0
Number of Embeds
5,108
Actions
Shares
0
Downloads
45
Comments
1
Likes
16
Embeds 0
No embeds

No notes for slide

Keynote Digital Data Tips Tuesday - Amsterdam - June 24th 2014 - A/B-testing analytics errors

  1. 1. < H a v e m o r e f u n A / B - t e s t i n g > A/B-t e s t i n g Errors To n We s s e l i n g – C EO Te s t i n g . A g e n c y #DDTT A m s t e r d a m 2 0 1 4 / 0 6 / 2 4
  2. 2. Why me? ü 10 years+ A/B-test experience ü I always loved the numbers side ü I’ve made all these mistakes Email: ton@testing.agency Twitter: @TonW #DDTT
  3. 3. Email: ton@testing.agency Twitter: @TonW #DDTT
  4. 4. We optimize Email: ton@testing.agency Twitter: @TonW #DDTT
  5. 5. Nature Email: ton@testing.agency Twitter: @TonW #DDTT
  6. 6. A better example Email: ton@testing.agency Twitter: @TonW #DDTT
  7. 7. Groundhog Day – The A/B-test movie Email: ton@testing.agency Twitter: @TonW #DDTT
  8. 8. Trying to get the girl Email: ton@testing.agency Twitter: @TonW #DDTT
  9. 9. Constant testing environment A / B Testing Email: ton@testing.agency Twitter: @TonW #DDTT
  10. 10. Better Dialogue Optimization Default Variation B Variation Winner!" A Email: ton@testing.agency Twitter: @TonW #DDTT
  11. 11. This is why you’re A/B-testing Small steps to learn how to make the next big step Email: ton@testing.agency Twitter: @TonW #DDTT
  12. 12. Testing is not free Use it to LEARN Email: ton@testing.agency Twitter: @TonW #DDTT
  13. 13. 50% of the A/B-tests I get to see is wrong Email: ton@testing.agency Twitter: @TonW #DDTT
  14. 14. Look at your A/B-testing results! +24% +52% +126% +83% +12% Email: ton@testing.agency Twitter: @TonW #DDTT +16%
  15. 15. Did you push the winners live to your site? You must be rich now! Email: ton@testing.agency Twitter: @TonW #DDTT
  16. 16. Or is this reality? Email: ton@testing.agency Twitter: @TonW #DDTT
  17. 17. Or even worse: A/B-test winners are letting you down! Email: ton@testing.agency Twitter: @TonW #DDTT
  18. 18. Are these real winners? +24% +52% +83% +12% +126% Email: ton@testing.agency Twitter: @TonW #DDTT +16%
  19. 19. 20 reasons why Your A/B-test analyzing sucks Email: ton@testing.agency Twitter: @TonW #DDTT
  20. 20. Get your statistics right! Email: ton@testing.agency Twitter: @TonW #DDTT
  21. 21. 1: low significance levels ü Big chance the winner is not a real winner Email: ton@testing.agency Twitter: @TonW #DDTT
  22. 22. 2: low power levels ü Big chance a real winner is not recognized Email: ton@testing.agency Twitter: @TonW #DDTT
  23. 23. Example test set-up ü 90% significance level ü 100 conversions per variation ü Average conversion rate: 2% ü 20 out of 100 ideas is a winner ü With an average uplift of 10% Email: ton@testing.agency Twitter: @TonW #DDTT
  24. 24. Average: 75% significance & 40% power Email: ton@testing.agency Twitter: @TonW #DDTT
  25. 25. So: ü 100 tests will give 10 false positives (90% significance) ü 40% of the 20 winners are recognized (40% power) ü 8 real winners & 10 false positives ü NOT 18 improvements of 15%+ Email: ton@testing.agency Twitter: @TonW #DDTT
  26. 26. 3: Not calculating traffic up front ü You thought the page had thousands of visitors Email: ton@testing.agency Twitter: @TonW #DDTT
  27. 27. 4: Not calculating what you can improve ü You thought the page had lots and lots conversions Email: ton@testing.agency Twitter: @TonW #DDTT
  28. 28. ABTestGuide.com/calc Email: ton@testing.agency Twitter: @TonW #DDTT
  29. 29. 5: Not choosing your segments up front ü You just go and dig unitil you find something (you will always find something that seems true BUT is not!) Email: ton@testing.agency Twitter: @TonW #DDTT
  30. 30. 6: You stop the test once it is significant Email: ton@testing.agency Twitter: @TonW #DDTT
  31. 31. 6: You stop the test once it is significant Email: ton@testing.agency Twitter: @TonW #DDTT
  32. 32. 7: You just let it run until it’s significant Email: ton@testing.agency Twitter: @TonW #DDTT
  33. 33. 7: You just let it run until it’s significant Your test should have a fixed lenght! Email: ton@testing.agency Twitter: @TonW #DDTT
  34. 34. 8: Your test takes too long ü Cookie deletion (you will loose 10% in two weeks) They will re-enter the the and pollute your samples! Email: ton@testing.agency Twitter: @TonW #DDTT
  35. 35. 9: You just choose a test length ü At least 1 full purchase cycle! Email: ton@testing.agency Twitter: @TonW #DDTT
  36. 36. 10: You don’t test full weeks ü At least 1 full week (if your not testing for time of day / week effects) Email: ton@testing.agency Twitter: @TonW #DDTT
  37. 37. 11: You think they always use one decive ü Only test 1 device at a time! Email: ton@testing.agency Twitter: @TonW #DDTT
  38. 38. 12: You’re testing broken stuff ü Always check browser compatibility! ü Dont’ break dynamic stuff! Email: ton@testing.agency Twitter: @TonW #DDTT
  39. 39. 13: You have multiple tests that overlap ü Your variation did not won (it was a combination of variations) Email: ton@testing.agency Twitter: @TonW #DDTT
  40. 40. 14: You thought this page matters ü But it did not (so you decide to use some other metric to declare the winner) Email: ton@testing.agency Twitter: @TonW #DDTT
  41. 41. 15: You forgot to leave out some visitors ü Only test on those who can be changed (if you test a sales promotion – leave out your current clients who just logon) Email: ton@testing.agency Twitter: @TonW #DDTT
  42. 42. 16: You’re not slowing the control down ü Add the same code to the control Email: ton@testing.agency Twitter: @TonW #DDTT
  43. 43. 17: You’re not testing on fresh people ü Start with control for a full purchase cycle (and then start sending traffic to your variations – setting the control to 0%) Email: ton@testing.agency Twitter: @TonW #DDTT
  44. 44. 18: You forgot to give them time to buy ü Stop getting fresh visitors after your test time (send new traffic to the control, but give you tested people time to finish converting) Email: ton@testing.agency Twitter: @TonW #DDTT
  45. 45. 19: You’re not diving into your analytics ü Always measure with analytics software (You will find conversions that are not supposed to be there!) Email: ton@testing.agency Twitter: @TonW #DDTT
  46. 46. 20: You’re not measuring the right stuff ü Use unique visitors ü Against lifetime value predictors! Email: ton@testing.agency Twitter: @TonW #DDTT
  47. 47. If you keep on making mistakes: Stuff like this happens Email: ton@testing.agency Twitter: @TonW #DDTT
  48. 48. Testing is not free Use it to LEARN Email: ton@testing.agency Twitter: @TonW #DDTT
  49. 49. Bonus: You did not re-test your winners ü Always re-test to be sure it was not a glitch Email: ton@testing.agency Twitter: @TonW #DDTT
  50. 50. So when you implement winners This will happen! Email: ton@testing.agency Twitter: @TonW #DDTT
  51. 51. < H a v e m o r e f u n A / B - t e s t i n g > A/B-t e s t i n g Errors To n We s s e l i n g – Te s t Wi n n e r #DDTT A m s t e r d a m 2 0 1 4 / 0 6 / 2 4
  52. 52. ABTestGuide.com/calc Email: ton@testing.agency Twitter: @TonW #DDTT

×