Uploaded byAleksandr Osiyuk
A/B testing for startups
Author https://t.me/BigQuery A/B testing for startups: https://link.medium.com/yANSP9F9I2 Startup must run A/B tests another ways that large companies
![ict_presentation_final_final_final[1].pptx](https://cdn.slidesharecdn.com/ss_thumbnails/ictpresentationfinalfinalfinal1-251230145259-2b4839bd-thumbnail.jpg?width=640&height=640&fit=bounds)
A/B testing for startups
- 1. A/B Testing forStartups by Aleksandr Osiyuk Product Analyst at MacPaw t.me/osiyuk
- 2. MacPaw products CleanMyMac SetappGemini Gemini Photos Clean External Drives to Get More Space The first subscription store for Mac apps Wipe Duplicate Files Off Your Mac Optimize iPhone photo storage
- 3. Startup must runA/B tests another ways that large companies
- 4. Analyst & PMexpectations
- 5. Analyst & PMexpectations Analyst needs to: ● apply a scientific approach ● dig into metrics ● try different methods of analysis
- 6. Analyst & PMexpectations Analyst needs to: ● apply a scientific approach ● dig into metrics ● try different methods of analysis PM needs to: ● fix a winner as soon as possible ● know how much money it produces
- 7. Factors affected thetesting time
- 8. Factors affected thetesting time ● Base conversion rate
- 9. Factors affected thetesting time ● Base conversion rate ● Relative increase in conversion rate
- 10. Factors affected thetesting time ● Base conversion rate ● Relative increase in conversion rate ● Daily number of users
- 11. Base CR Relative increase New CRSample size Case 1 1% 5% 1.05% 626,230 Case 2 2% 5% 2.1% 309,927 Case 3 3% 5% 3.2% 204,493 Case 4 4% 5% 4.2% 151,776 Case 5 5% 5% 5.3% 120,145 Case 6 10% 5% 10.5% 56,885 Case 7 15% 5% 15.8% 35,798 Case 8 20% 5% 21.0% 25,255 Case 9 25% 5% 26.3% 18,929
- 12. Base CR Relative increase New CRSample size Case 1 10% 1% 10.1% 1,414,680 Case 2 10% 2% 10.2% 354,138 Case 3 10% 3% 10.3% 157,602 Case 4 10% 4% 10.4% 88,767 Case 5 10% 5% 10.5% 56,885 Case 6 10% 10% 11.0% 14,312 Case 7 10% 15% 11.5% 6,401 Case 8 10% 20% 12.0% 3,622 Case 9 10% 25% 12.5% 2,332
- 13. Tips to speedup testing
- 14. Tips to speedup testing ● Use metric that is closely correlated to final conversion
- 15. Tips to speedup testing ● Use metric that is closely correlated to final conversion ● Testing big changes
- 16. Tips to speedup testing ● Use metric that is closely correlated to final conversion ● Testing big changes ● Divide users in the middle of a funnel
- 17. CR from 1step to 3 step CR from 2 step to 3 step Base CR: 50/1000=5% Relative increase: 10% Sample size: 30 244 (30 days) 1 step 1000 2 step 200 Target action 55 5 50Base CR: 50/200=25% Relative increase: 10% Sample size: 4 753 (24 days) CR 25% CR 5%
- 18. CR from 1step to 3 step CR from 2 step to 3 step Base CR: 50/1000=5% Relative increase: 10% Sample size: 30 244 (30 days) 1 step 1000 2 step 200 55 5 50Base CR: 50/200=25% Relative increase: 10% Sample size: 4 753 (24 days) CR 25% CR 5% A/A test Target action
- 19. Tips to speedup testing ● Use metric that is closely correlated to final conversion ● Testing big changes ● Divide users in the middle of a funnel ● Increase visual contrast to be noticed by users
- 20. Tips to speedup testing Users every day Actions CR Relative increase Sample size n, days 1000 120 12% 10% 11657 11.7
- 21. Tips to speedup testing Users every day Actions CR Relative increase Sample size n, days 1000 120 12% 10% 11657 11.7 500 50 10% 8%
- 22. Tips to speedup testing Users every day Actions CR Relative increase Sample size n, days 1000 120 12% 10% 11657 11.7 500 50 10% 8% 1500 170 11.33% 9.41% 14041 9.4
- 23. Tips to speedup testing CR Relativeincrease
- 24. Tips to speedup testing ● Use metric that is closely correlated to final conversion ● Testing big changes ● Divide users in the middle of a funnel ● Increase visual contrast to be noticed by users ● Calculate minimum detectable effect (MDE)
- 25. Another ways tospeed up testing
- 26. Another ways tospeed up testing ● Metrics: binary / non-binary
- 27. Another ways tospeed up testing ● Metrics: binary / non-binary ● Analysis: bayesian / frequentist
- 28. Another ways tospeed up testing ● Metrics: binary / non-binary ● Analysis: bayesian / frequentist ● Hypotheses: innovate / optimize
- 29.
- 30.
- 31. Experiments with monetization Printmagazine - 20$ Digital magazine - 10$
- 32. Print magazine -20$ Digital magazine - 10$ 1 29% 2 71% AOV: 13$
- 33. Experiments with monetization Printmagazine - 20$ Digital magazine - 10$ Print & Digital - 20$
- 34. Print magazine -20$ Digital magazine - 10$ Print & Digital - 20$ 1 11% 2 30% 3 59% AOV: 17$
- 35. Experiments with monetization Digitalmagazine - 10$ Print & Digital - 20$
- 36. Digital magazine -10$ Print & Digital - 20$ 1 61% 2 39% AOV: 14$
- 37. A/B testing forInsights
- 38. A/B testing forInsights ● Check more metrics and user flow
- 39. A/B testing forInsights ● Check more metrics and user flow ● Segmentation (old/new, traffic, country, behavior etc.)
- 40. A/B testing forInsights ● Check more metrics and user flow ● Segmentation (old/new, traffic, country, behavior etc.) ● Mix overlapping / chained tests
- 41. A/B testing forInsights ● Check more metrics and user flow ● Segmentation (old/new, traffic, country, behavior etc.) ● Mix overlapping / chained tests ● Multiple comparisons problem
- 42. A/B testing forInsights ● Check more metrics and user flow ● Segmentation (old/new, traffic, country, behavior etc.) ● Mix overlapping / chained tests ● Multiple comparisons problem ● Simpson’s paradox
- 43. Male Female 568 of2000 (28%) 372 of 2000 (19%) Simpson’s paradox
- 44. Male Female 568 of2000 (28%) 372 of 2000 (19%) Highly competitive subjects 168 of 1200 (14%) 270 of 1800 (15%) Lowly competitive subjects 400 of 800 (50%) 102 of 200 (51%) Simpson’s paradox
- 45. А/В testing featuresin apps
- 46. А/В testing featuresin apps ● Why do you need to split the traffic on server-side? ● If there is no internet? (А/А-test in apps 50/50? Case about custom settings)
- 47. Calculator: bit.ly/2JswQWc t.me/BigQueryThanks forattention! Telegram channel about А/В-testing: t.me/ABtesting