Downloaded 20 times
Uploaded byWebanalisten .nl
DDTT11: Ton Wesseling - 21-01-20
This document outlines the key tasks of an analyst within an A/B testing culture. The three main tasks are: 1) Ensuring high quality data is available, 2) Prioritizing experiments that have the highest potential for effectiveness, and 3) Calculating the real financial impact of experiments including return on investment. The analyst helps maximize growth by focusing experiments in areas with the highest ROI while minimizing costs like false discovery rates.
More Related Content
Similar to DDTT11: Ton Wesseling - 21-01-20
DDTT11: Ton Wesseling - 21-01-20
- 1. Ton Wesseling How ananalyst can add value! Digital Experiments
- 2.
- 3.
- 4.
- 5. TON@ONLINEDIALOGUE.COM A/B-tes5ng mastery course This talk onlymakes sense if you have 10.000 transactions or more per month – enough to get experimentation in the DNA of your organization.!
- 6.
- 7.
- 8. TON@ONLINEDIALOGUE.COM DEF! The task ofan analyst within an A/B-testing Culture! 1. Data! 2. Effectiveness! 3. Finance!
- 9. TON@ONLINEDIALOGUE.COM Data! Let there behigh quality data!
- 10. TON@ONLINEDIALOGUE.COM Make sure allfunnels are measured…!
- 11. TON@ONLINEDIALOGUE.COM Make sure yourtesting solution has all users! Users on template: 42186! Users in the tool: 37652! Users with code executed: 34312 ! 100%! 89%! 81%!
- 12. TON@ONLINEDIALOGUE.COM What if myexperiments had 20% more users?!
- 13.
- 14. TON@ONLINEDIALOGUE.COM Be able tosegment on page interactions!
- 15. TON@ONLINEDIALOGUE.COM Be able tosegment on who can be influenced!
- 16. TON@ONLINEDIALOGUE.COM Be able tosegment on who can be influenced!
- 17. TON@ONLINEDIALOGUE.COM Be able tocreate behavioral segments! Typical ecommerce flow example: ü All users on your website with enough time to take action ü All users on your website with at least some interaction ü All users on your website with heavy interaction ü All users on your website with clear intent to buy ü All users on your website that are willing to buy ü All users on your website that succeed in buying ü All users on your website that return with intent to buy more Funnel + Average 5me
- 18. TON@ONLINEDIALOGUE.COM Data! Let there behigh quality data!
- 19. TON@ONLINEDIALOGUE.COM DEF! The task ofan analyst within an A/B-testing Culture! 1. Data! 2. Effectiveness! 3. Finance!
- 20. TON@ONLINEDIALOGUE.COM Effectiveness! Make sure youwork on stuff! with the highest potential outcome!
- 21. TON@ONLINEDIALOGUE.COM Statistical Power! The likelihoodthat an experiment will detect an effect, when there is an effect there to be detected!
- 22. TON@ONLINEDIALOGUE.COM Power & Significance Newversion is NOT better New version is better New version is NOT better New version is better Measured Reality
- 23. TON@ONLINEDIALOGUE.COM Power & Significance Newversion is NOT better New version is better New version is NOT better New version is better Measured Reality
- 24. TON@ONLINEDIALOGUE.COM Power & Significance Newversion is NOT better New version is better New version is NOT better New version is better Measured Reality
- 25. TON@ONLINEDIALOGUE.COM Power & Significance Donot reject H0 Reject H0 New version is NOT better New version is better Measured Reality
- 26. TON@ONLINEDIALOGUE.COM Power & Significance Donot reject H0 Reject H0 H0 is true H0 is false Measured Reality
- 27. TON@ONLINEDIALOGUE.COM Significance Do not rejectH0 Reject H0 H0 is true H0 is false Correct decision J Measured Reality
- 28. TON@ONLINEDIALOGUE.COM Significance Do not rejectH0 Reject H0 H0 is true Type I False Positive (α) H0 is false Correct decision J Measured Reality
- 29. TON@ONLINEDIALOGUE.COM Power Do not rejectH0 Reject H0 H0 is true Correct decision J Type I False Positive (α) H0 is false Correct decision J Measured Reality
- 30. TON@ONLINEDIALOGUE.COM Power Do not rejectH0 Reject H0 H0 is true Correct decision J Type I False Positive (α) H0 is false Type II False Negative (β) Correct decision J Measured Reality
- 31. TON@ONLINEDIALOGUE.COM Power New version is NOTbetter New version is better New version is NOT better Correct decision J Type I False Positive (α) New version is better Type II False Negative (β) Correct decision J Measured Reality
- 32. TON@ONLINEDIALOGUE.COM Power & Significancerule of thumb Power When you start: try to test on pages with a high Power (>80%) à otherwise you don’t detect effects when there is an effect to be detected (False negatives). Significance When you start: try to test against a high enough significance level (90%) à otherwise you’ll declare winners, when in reality there isn’t an effect (False positives).
- 33.
- 34.
- 35.
- 36. TON@ONLINEDIALOGUE.COM Prioritize based onMDE to start!
- 37. TON@ONLINEDIALOGUE.COM Prioritize based onmeasured results!!
- 38. TON@ONLINEDIALOGUE.COM Effectiveness! Make sure youwork on stuff! with the highest potential outcome!
- 39. TON@ONLINEDIALOGUE.COM DEF! The task ofan analyst within an A/B-testing Culture! 1. Data! 2. Effectiveness! 3. Finance!
- 40.
- 41. TON@ONLINEDIALOGUE.COM What does yourcalculation look like?! If significant result:! ! Extra new customers per week! *! 52 weeks effective! *! Average lifetime value!
- 42. TON@ONLINEDIALOGUE.COM So this experimentwill bring us:! €412.390!
- 43.
- 44.
- 45. TON@ONLINEDIALOGUE.COM So this experimentwill bring us?! €412.390 * (100%-Type-M error %)?!
- 46. TON@ONLINEDIALOGUE.COM Prioritize based onmeasured results?! * (100% - M-Type Error) of course!
- 47. TON@ONLINEDIALOGUE.COM What is yourfalse discovery rate?! Significance border: 90%! 100 experiments! 20 significant outcomes! ! 50%!* (it’s a little lower, this is the poor man’s calculation)! (with every real win the number of experiments without wins becomes lower, which leads to less false positives)!
- 48. TON@ONLINEDIALOGUE.COM So not really50%! FDR* = (Measured Wins - ((Measured Wins - ! ((100% - Confidence Level) * Experiments))! / Confidence Level)) / Measured Wins! ! =! ! (20 – ((20 – ((100% - 90%) * 100)) / 90%)) / 20! ! =! ! 44%!* (only if your power on all experiments was 100%)! (Your Power will be lower, which means you had more real wins, but not measured (false negatives).! This leads to less experiments without an effect, so the number of false positives will be even lower)!
- 49.
- 50.
- 51. TON@ONLINEDIALOGUE.COM So all yourexperiments will bring you:! Sum of (every winner *! ! (100% - Type-M error % per winner))! *! (100% - FDR%)! *! Implementation % (within x months…)! (assuming every new win is tested on the new default where all earlier wins are implemented)!
- 52. TON@ONLINEDIALOGUE.COM You can correctFDR for P-value distribution!
- 53. TON@ONLINEDIALOGUE.COM So all yourexperiments will bring you:! Sum of (every winner *! ! (100% - Type-M error % per winner))! *! (100% - corrected FDR%)! *! Implementation % (within x months…)! (assuming every new win is tested on the new default where all earlier wins are implemented)!
- 54. TON@ONLINEDIALOGUE.COM Maximize your growthwith ROI limits:! Value of A/B-testing for Optimization! ___________________________________! ! Costs of A/B-testing for Optimization! = ROI!
- 55. TON@ONLINEDIALOGUE.COM Finance: are youabove or below your ROI limit?! 1. Above: increase budgets! 2. Below: increase knowledge! 3. Still below: decrease budgets!
- 56.
- 57. TON@ONLINEDIALOGUE.COM DEF! The task ofan analyst within an A/B-testing Culture! 1. Data! 2. Effectiveness! 3. Finance!
- 58.
- 59. TON@ONLINEDIALOGUE.COM Data Analyst - The Noun Project icon from the Noun Project An A/B-testing forgrowth analyst:! 1. Makes sure there is high quality Data available! 2. Steers the data chance on Effect! 3. Reports on the real Financial impact!
- 60.
- 61. Ton Wesseling https://ondi.me/tonw Let’s connecton LinkedIn Latest article on A/B-testing:
- 62. Ton Wesseling How ananalyst can add value! Digital Experiments