Your SlideShare is downloading. ×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Keynote Digital Data Tips Tuesday - Amsterdam - June 24th 2014 - A/B-testing analytics errors

6,661
views

Published on

Keynote by @tonw / Ton Wesseling - CEO of http://testing.agency - he shares all the failures he saw and also made himself on analytics and A/B-testing. …

Keynote by @tonw / Ton Wesseling - CEO of http://testing.agency - he shares all the failures he saw and also made himself on analytics and A/B-testing.

Are you also questioning why your total website revenue is not exploding after you've put all those A/B-test winners live? Find the answers in these slides.

Published in: Marketing, Technology, Business

2 Comments
14 Likes
Statistics
Notes
  • We will help you perform a proper A/B testing, generate hypotheses and ideas for you, install all needed tools and run tests. Learn more at http://abtst.com/agency?lang=en&utm_source=slideshare&utm_medium=comment&utm_campaign=comments
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Please check out the AB-test significance, power and minimum sample size calculator mentioned in this presentation at http://abtestguide.com/calc
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
6,661
On Slideshare
0
From Embeds
0
Number of Embeds
20
Actions
Shares
0
Downloads
36
Comments
2
Likes
14
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. < H a v e m o r e f u n A / B - t e s t i n g > A/B-t e s t i n g Errors To n We s s e l i n g – C EO Te s t i n g . A g e n c y #DDTT A m s t e r d a m 2 0 1 4 / 0 6 / 2 4
  • 2. Why me? ü 10 years+ A/B-test experience ü I always loved the numbers side ü I’ve made all these mistakes Email: ton@testing.agency Twitter: @TonW #DDTT
  • 3. Email: ton@testing.agency Twitter: @TonW #DDTT
  • 4. We optimize Email: ton@testing.agency Twitter: @TonW #DDTT
  • 5. Nature Email: ton@testing.agency Twitter: @TonW #DDTT
  • 6. A better example Email: ton@testing.agency Twitter: @TonW #DDTT
  • 7. Groundhog Day – The A/B-test movie Email: ton@testing.agency Twitter: @TonW #DDTT
  • 8. Trying to get the girl Email: ton@testing.agency Twitter: @TonW #DDTT
  • 9. Constant testing environment A / B Testing Email: ton@testing.agency Twitter: @TonW #DDTT
  • 10. Better Dialogue Optimization Default Variation B Variation Winner!" A Email: ton@testing.agency Twitter: @TonW #DDTT
  • 11. This is why you’re A/B-testing Small steps to learn how to make the next big step Email: ton@testing.agency Twitter: @TonW #DDTT
  • 12. Testing is not free Use it to LEARN Email: ton@testing.agency Twitter: @TonW #DDTT
  • 13. 50% of the A/B-tests I get to see is wrong Email: ton@testing.agency Twitter: @TonW #DDTT
  • 14. Look at your A/B-testing results! +24% +52% +126% +83% +12% Email: ton@testing.agency Twitter: @TonW #DDTT +16%
  • 15. Did you push the winners live to your site? You must be rich now! Email: ton@testing.agency Twitter: @TonW #DDTT
  • 16. Or is this reality? Email: ton@testing.agency Twitter: @TonW #DDTT
  • 17. Or even worse: A/B-test winners are letting you down! Email: ton@testing.agency Twitter: @TonW #DDTT
  • 18. Are these real winners? +24% +52% +83% +12% +126% Email: ton@testing.agency Twitter: @TonW #DDTT +16%
  • 19. 20 reasons why Your A/B-test analyzing sucks Email: ton@testing.agency Twitter: @TonW #DDTT
  • 20. Get your statistics right! Email: ton@testing.agency Twitter: @TonW #DDTT
  • 21. 1: low significance levels ü Big chance the winner is not a real winner Email: ton@testing.agency Twitter: @TonW #DDTT
  • 22. 2: low power levels ü Big chance a real winner is not recognized Email: ton@testing.agency Twitter: @TonW #DDTT
  • 23. Example test set-up ü 90% significance level ü 100 conversions per variation ü Average conversion rate: 2% ü 20 out of 100 ideas is a winner ü With an average uplift of 10% Email: ton@testing.agency Twitter: @TonW #DDTT
  • 24. Average: 75% significance & 40% power Email: ton@testing.agency Twitter: @TonW #DDTT
  • 25. So: ü 100 tests will give 10 false positives (90% significance) ü 40% of the 20 winners are recognized (40% power) ü 8 real winners & 10 false positives ü NOT 18 improvements of 15%+ Email: ton@testing.agency Twitter: @TonW #DDTT
  • 26. 3: Not calculating traffic up front ü You thought the page had thousands of visitors Email: ton@testing.agency Twitter: @TonW #DDTT
  • 27. 4: Not calculating what you can improve ü You thought the page had lots and lots conversions Email: ton@testing.agency Twitter: @TonW #DDTT
  • 28. ABTestGuide.com/calc Email: ton@testing.agency Twitter: @TonW #DDTT
  • 29. 5: Not choosing your segments up front ü You just go and dig unitil you find something (you will always find something that seems true BUT is not!) Email: ton@testing.agency Twitter: @TonW #DDTT
  • 30. 6: You stop the test once it is significant Email: ton@testing.agency Twitter: @TonW #DDTT
  • 31. 6: You stop the test once it is significant Email: ton@testing.agency Twitter: @TonW #DDTT
  • 32. 7: You just let it run until it’s significant Email: ton@testing.agency Twitter: @TonW #DDTT
  • 33. 7: You just let it run until it’s significant Your test should have a fixed lenght! Email: ton@testing.agency Twitter: @TonW #DDTT
  • 34. 8: Your test takes too long ü Cookie deletion (you will loose 10% in two weeks) They will re-enter the the and pollute your samples! Email: ton@testing.agency Twitter: @TonW #DDTT
  • 35. 9: You just choose a test length ü At least 1 full purchase cycle! Email: ton@testing.agency Twitter: @TonW #DDTT
  • 36. 10: You don’t test full weeks ü At least 1 full week (if your not testing for time of day / week effects) Email: ton@testing.agency Twitter: @TonW #DDTT
  • 37. 11: You think they always use one decive ü Only test 1 device at a time! Email: ton@testing.agency Twitter: @TonW #DDTT
  • 38. 12: You’re testing broken stuff ü Always check browser compatibility! ü Dont’ break dynamic stuff! Email: ton@testing.agency Twitter: @TonW #DDTT
  • 39. 13: You have multiple tests that overlap ü Your variation did not won (it was a combination of variations) Email: ton@testing.agency Twitter: @TonW #DDTT
  • 40. 14: You thought this page matters ü But it did not (so you decide to use some other metric to declare the winner) Email: ton@testing.agency Twitter: @TonW #DDTT
  • 41. 15: You forgot to leave out some visitors ü Only test on those who can be changed (if you test a sales promotion – leave out your current clients who just logon) Email: ton@testing.agency Twitter: @TonW #DDTT
  • 42. 16: You’re not slowing the control down ü Add the same code to the control Email: ton@testing.agency Twitter: @TonW #DDTT
  • 43. 17: You’re not testing on fresh people ü Start with control for a full purchase cycle (and then start sending traffic to your variations – setting the control to 0%) Email: ton@testing.agency Twitter: @TonW #DDTT
  • 44. 18: You forgot to give them time to buy ü Stop getting fresh visitors after your test time (send new traffic to the control, but give you tested people time to finish converting) Email: ton@testing.agency Twitter: @TonW #DDTT
  • 45. 19: You’re not diving into your analytics ü Always measure with analytics software (You will find conversions that are not supposed to be there!) Email: ton@testing.agency Twitter: @TonW #DDTT
  • 46. 20: You’re not measuring the right stuff ü Use unique visitors ü Against lifetime value predictors! Email: ton@testing.agency Twitter: @TonW #DDTT
  • 47. If you keep on making mistakes: Stuff like this happens Email: ton@testing.agency Twitter: @TonW #DDTT
  • 48. Testing is not free Use it to LEARN Email: ton@testing.agency Twitter: @TonW #DDTT
  • 49. Bonus: You did not re-test your winners ü Always re-test to be sure it was not a glitch Email: ton@testing.agency Twitter: @TonW #DDTT
  • 50. So when you implement winners This will happen! Email: ton@testing.agency Twitter: @TonW #DDTT
  • 51. < H a v e m o r e f u n A / B - t e s t i n g > A/B-t e s t i n g Errors To n We s s e l i n g – Te s t Wi n n e r #DDTT A m s t e r d a m 2 0 1 4 / 0 6 / 2 4
  • 52. ABTestGuide.com/calc Email: ton@testing.agency Twitter: @TonW #DDTT