< H a v e m o r e f u n A / B - t e s t i n g > 
A/B-t e s t i n g Errors 
To n We s s e l i n g – C EO Te s t i n g . A g...
Why me? 
ü 10 years+ A/B-test experience 
ü I always loved the numbers side 
ü I’ve made all these mistakes 
Email: ton...
Email: ton@testing.agency Twitter: @TonW #DDTT
We optimize 
Email: ton@testing.agency Twitter: @TonW #DDTT
Nature 
Email: ton@testing.agency Twitter: @TonW #DDTT
A better example 
Email: ton@testing.agency Twitter: @TonW #DDTT
Groundhog Day – The A/B-test movie 
Email: ton@testing.agency Twitter: @TonW #DDTT
Trying to get the girl 
Email: ton@testing.agency Twitter: @TonW #DDTT
Constant testing environment 
A / B Testing 
Email: ton@testing.agency Twitter: @TonW #DDTT
Better Dialogue Optimization 
Default 
Variation 
B 
Variation 
Winner!" 
A 
Email: ton@testing.agency Twitter: @TonW #DDT...
This is why you’re A/B-testing 
Small steps to learn how 
to make the next big step 
Email: ton@testing.agency Twitter: @T...
Testing is not free 
Use it to LEARN 
Email: ton@testing.agency Twitter: @TonW #DDTT
50% of the A/B-tests I get to see is 
wrong 
Email: ton@testing.agency Twitter: @TonW #DDTT
Look at your A/B-testing results! 
+24% 
+52% 
+126% 
+83% 
+12% 
Email: ton@testing.agency Twitter: @TonW #DDTT 
+16%
Did you push the winners live to your 
site? 
You must be rich now! 
Email: ton@testing.agency Twitter: @TonW #DDTT
Or is this reality? 
Email: ton@testing.agency Twitter: @TonW #DDTT
Or even worse: 
A/B-test winners are letting you down! 
Email: ton@testing.agency Twitter: @TonW #DDTT
Are these real winners? 
+24% +52% 
+83% +12% +126% 
Email: ton@testing.agency Twitter: @TonW #DDTT 
+16%
20 reasons why 
Your A/B-test 
analyzing 
sucks 
Email: ton@testing.agency Twitter: @TonW #DDTT
Get your statistics right! 
Email: ton@testing.agency Twitter: @TonW #DDTT
1: low significance levels 
ü Big chance the winner is not a real winner 
Email: ton@testing.agency Twitter: @TonW #DDTT
2: low power levels 
ü Big chance a real winner is not recognized 
Email: ton@testing.agency Twitter: @TonW #DDTT
Example test set-up 
ü 90% significance level 
ü 100 conversions per variation 
ü Average conversion rate: 2% 
ü 20 ou...
Average: 75% significance & 40% power 
Email: ton@testing.agency Twitter: @TonW #DDTT
So: 
ü 100 tests will give 10 false positives (90% significance) 
ü 40% of the 20 winners are recognized (40% power) 
ü...
3: Not calculating traffic up front 
ü You thought the page had thousands of 
visitors 
Email: ton@testing.agency Twitter...
4: Not calculating what you can improve 
ü You thought the page had lots and lots 
conversions 
Email: ton@testing.agency...
ABTestGuide.com/calc 
Email: ton@testing.agency Twitter: @TonW #DDTT
5: Not choosing your segments up front 
ü You just go and dig unitil you find something 
(you will always find something ...
6: You stop the test once it is significant 
Email: ton@testing.agency Twitter: @TonW #DDTT
6: You stop the test once it is significant 
Email: ton@testing.agency Twitter: @TonW #DDTT
7: You just let it run until it’s significant 
Email: ton@testing.agency Twitter: @TonW #DDTT
7: You just let it run until it’s significant 
Your test should have a fixed lenght! 
Email: ton@testing.agency Twitter: @...
8: Your test takes too long 
ü Cookie deletion 
(you will loose 10% in two weeks) 
They will re-enter the the and pollute...
9: You just choose a test length 
ü At least 1 full purchase cycle! 
Email: ton@testing.agency Twitter: @TonW #DDTT
10: You don’t test full weeks 
ü At least 1 full week 
(if your not testing for time of day / week effects) 
Email: ton@t...
11: You think they always use one decive 
ü Only test 1 device at a time! 
Email: ton@testing.agency Twitter: @TonW #DDTT
12: You’re testing broken stuff 
ü Always check browser compatibility! 
ü Dont’ break dynamic stuff! 
Email: ton@testing...
13: You have multiple tests that overlap 
ü Your variation did not won 
(it was a combination of variations) 
Email: ton@...
14: You thought this page matters 
ü But it did not 
(so you decide to use some other metric to declare the winner) 
Emai...
15: You forgot to leave out some visitors 
ü Only test on those who can be changed 
(if you test a sales promotion – leav...
16: You’re not slowing the control down 
ü Add the same code to the control 
Email: ton@testing.agency Twitter: @TonW #DD...
17: You’re not testing on fresh people 
ü Start with control for a full purchase cycle 
(and then start sending traffic t...
18: You forgot to give them time to buy 
ü Stop getting fresh visitors after your test time 
(send new traffic to the con...
19: You’re not diving into your analytics 
ü Always measure with analytics software 
(You will find conversions that are ...
20: You’re not measuring the right stuff 
ü Use unique visitors 
ü Against lifetime value predictors! 
Email: ton@testin...
If you keep on making mistakes: 
Stuff like this happens 
Email: ton@testing.agency Twitter: @TonW #DDTT
Testing is not free 
Use it to LEARN 
Email: ton@testing.agency Twitter: @TonW #DDTT
Bonus: You did not re-test your winners 
ü Always re-test to be sure it was not a glitch 
Email: ton@testing.agency Twitt...
So when you implement winners 
This will happen! 
Email: ton@testing.agency Twitter: @TonW #DDTT
< H a v e m o r e f u n A / B - t e s t i n g > 
A/B-t e s t i n g Errors 
To n We s s e l i n g – Te s t Wi n n e r 
#DDT...
ABTestGuide.com/calc 
Email: ton@testing.agency Twitter: @TonW #DDTT
Upcoming SlideShare
Loading in...5
×

Keynote Digital Data Tips Tuesday - Amsterdam - June 24th 2014 - A/B-testing analytics errors

7,853

Published on

Keynote by @tonw / Ton Wesseling - CEO of http://testing.agency - he shares all the failures he saw and also made himself on analytics and A/B-testing.

Are you also questioning why your total website revenue is not exploding after you've put all those A/B-test winners live? Find the answers in these slides.

Published in: Marketing, Technology, Business
1 Comment
14 Likes
Statistics
Notes
No Downloads
Views
Total Views
7,853
On Slideshare
0
From Embeds
0
Number of Embeds
20
Actions
Shares
0
Downloads
40
Comments
1
Likes
14
Embeds 0
No embeds

No notes for slide

Keynote Digital Data Tips Tuesday - Amsterdam - June 24th 2014 - A/B-testing analytics errors

  1. 1. < H a v e m o r e f u n A / B - t e s t i n g > A/B-t e s t i n g Errors To n We s s e l i n g – C EO Te s t i n g . A g e n c y #DDTT A m s t e r d a m 2 0 1 4 / 0 6 / 2 4
  2. 2. Why me? ü 10 years+ A/B-test experience ü I always loved the numbers side ü I’ve made all these mistakes Email: ton@testing.agency Twitter: @TonW #DDTT
  3. 3. Email: ton@testing.agency Twitter: @TonW #DDTT
  4. 4. We optimize Email: ton@testing.agency Twitter: @TonW #DDTT
  5. 5. Nature Email: ton@testing.agency Twitter: @TonW #DDTT
  6. 6. A better example Email: ton@testing.agency Twitter: @TonW #DDTT
  7. 7. Groundhog Day – The A/B-test movie Email: ton@testing.agency Twitter: @TonW #DDTT
  8. 8. Trying to get the girl Email: ton@testing.agency Twitter: @TonW #DDTT
  9. 9. Constant testing environment A / B Testing Email: ton@testing.agency Twitter: @TonW #DDTT
  10. 10. Better Dialogue Optimization Default Variation B Variation Winner!" A Email: ton@testing.agency Twitter: @TonW #DDTT
  11. 11. This is why you’re A/B-testing Small steps to learn how to make the next big step Email: ton@testing.agency Twitter: @TonW #DDTT
  12. 12. Testing is not free Use it to LEARN Email: ton@testing.agency Twitter: @TonW #DDTT
  13. 13. 50% of the A/B-tests I get to see is wrong Email: ton@testing.agency Twitter: @TonW #DDTT
  14. 14. Look at your A/B-testing results! +24% +52% +126% +83% +12% Email: ton@testing.agency Twitter: @TonW #DDTT +16%
  15. 15. Did you push the winners live to your site? You must be rich now! Email: ton@testing.agency Twitter: @TonW #DDTT
  16. 16. Or is this reality? Email: ton@testing.agency Twitter: @TonW #DDTT
  17. 17. Or even worse: A/B-test winners are letting you down! Email: ton@testing.agency Twitter: @TonW #DDTT
  18. 18. Are these real winners? +24% +52% +83% +12% +126% Email: ton@testing.agency Twitter: @TonW #DDTT +16%
  19. 19. 20 reasons why Your A/B-test analyzing sucks Email: ton@testing.agency Twitter: @TonW #DDTT
  20. 20. Get your statistics right! Email: ton@testing.agency Twitter: @TonW #DDTT
  21. 21. 1: low significance levels ü Big chance the winner is not a real winner Email: ton@testing.agency Twitter: @TonW #DDTT
  22. 22. 2: low power levels ü Big chance a real winner is not recognized Email: ton@testing.agency Twitter: @TonW #DDTT
  23. 23. Example test set-up ü 90% significance level ü 100 conversions per variation ü Average conversion rate: 2% ü 20 out of 100 ideas is a winner ü With an average uplift of 10% Email: ton@testing.agency Twitter: @TonW #DDTT
  24. 24. Average: 75% significance & 40% power Email: ton@testing.agency Twitter: @TonW #DDTT
  25. 25. So: ü 100 tests will give 10 false positives (90% significance) ü 40% of the 20 winners are recognized (40% power) ü 8 real winners & 10 false positives ü NOT 18 improvements of 15%+ Email: ton@testing.agency Twitter: @TonW #DDTT
  26. 26. 3: Not calculating traffic up front ü You thought the page had thousands of visitors Email: ton@testing.agency Twitter: @TonW #DDTT
  27. 27. 4: Not calculating what you can improve ü You thought the page had lots and lots conversions Email: ton@testing.agency Twitter: @TonW #DDTT
  28. 28. ABTestGuide.com/calc Email: ton@testing.agency Twitter: @TonW #DDTT
  29. 29. 5: Not choosing your segments up front ü You just go and dig unitil you find something (you will always find something that seems true BUT is not!) Email: ton@testing.agency Twitter: @TonW #DDTT
  30. 30. 6: You stop the test once it is significant Email: ton@testing.agency Twitter: @TonW #DDTT
  31. 31. 6: You stop the test once it is significant Email: ton@testing.agency Twitter: @TonW #DDTT
  32. 32. 7: You just let it run until it’s significant Email: ton@testing.agency Twitter: @TonW #DDTT
  33. 33. 7: You just let it run until it’s significant Your test should have a fixed lenght! Email: ton@testing.agency Twitter: @TonW #DDTT
  34. 34. 8: Your test takes too long ü Cookie deletion (you will loose 10% in two weeks) They will re-enter the the and pollute your samples! Email: ton@testing.agency Twitter: @TonW #DDTT
  35. 35. 9: You just choose a test length ü At least 1 full purchase cycle! Email: ton@testing.agency Twitter: @TonW #DDTT
  36. 36. 10: You don’t test full weeks ü At least 1 full week (if your not testing for time of day / week effects) Email: ton@testing.agency Twitter: @TonW #DDTT
  37. 37. 11: You think they always use one decive ü Only test 1 device at a time! Email: ton@testing.agency Twitter: @TonW #DDTT
  38. 38. 12: You’re testing broken stuff ü Always check browser compatibility! ü Dont’ break dynamic stuff! Email: ton@testing.agency Twitter: @TonW #DDTT
  39. 39. 13: You have multiple tests that overlap ü Your variation did not won (it was a combination of variations) Email: ton@testing.agency Twitter: @TonW #DDTT
  40. 40. 14: You thought this page matters ü But it did not (so you decide to use some other metric to declare the winner) Email: ton@testing.agency Twitter: @TonW #DDTT
  41. 41. 15: You forgot to leave out some visitors ü Only test on those who can be changed (if you test a sales promotion – leave out your current clients who just logon) Email: ton@testing.agency Twitter: @TonW #DDTT
  42. 42. 16: You’re not slowing the control down ü Add the same code to the control Email: ton@testing.agency Twitter: @TonW #DDTT
  43. 43. 17: You’re not testing on fresh people ü Start with control for a full purchase cycle (and then start sending traffic to your variations – setting the control to 0%) Email: ton@testing.agency Twitter: @TonW #DDTT
  44. 44. 18: You forgot to give them time to buy ü Stop getting fresh visitors after your test time (send new traffic to the control, but give you tested people time to finish converting) Email: ton@testing.agency Twitter: @TonW #DDTT
  45. 45. 19: You’re not diving into your analytics ü Always measure with analytics software (You will find conversions that are not supposed to be there!) Email: ton@testing.agency Twitter: @TonW #DDTT
  46. 46. 20: You’re not measuring the right stuff ü Use unique visitors ü Against lifetime value predictors! Email: ton@testing.agency Twitter: @TonW #DDTT
  47. 47. If you keep on making mistakes: Stuff like this happens Email: ton@testing.agency Twitter: @TonW #DDTT
  48. 48. Testing is not free Use it to LEARN Email: ton@testing.agency Twitter: @TonW #DDTT
  49. 49. Bonus: You did not re-test your winners ü Always re-test to be sure it was not a glitch Email: ton@testing.agency Twitter: @TonW #DDTT
  50. 50. So when you implement winners This will happen! Email: ton@testing.agency Twitter: @TonW #DDTT
  51. 51. < H a v e m o r e f u n A / B - t e s t i n g > A/B-t e s t i n g Errors To n We s s e l i n g – Te s t Wi n n e r #DDTT A m s t e r d a m 2 0 1 4 / 0 6 / 2 4
  52. 52. ABTestGuide.com/calc Email: ton@testing.agency Twitter: @TonW #DDTT
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×