A/B Testing in Plain English by eBay Product Manager
•
0 likes•428 views
Main takeaways: - Learn how to A/B test without having to know advanced statistics - Terminologies and concepts explained - Learn how to apply best practices and avoid common pitfalls to run insightful experiments
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to A/B Testing in Plain English by eBay Product Manager
Similar to A/B Testing in Plain English by eBay Product Manager (20)
More from Product School
More from Product School (20)
Recently uploaded
Recently uploaded (20)
A/B Testing in Plain English by eBay Product Manager
- 1. www.productschool.com A/B Testing in Plain English by eBay Product Manager
- 2. CERTIFICATES Your Product Management Certificate Path Product Leadership Certificate™ Full Stack Product Management Certificate™ Product Management Certificate™ 20 HOURS40 HOURS40 HOURS
- 3. Corporate Training Level up your team’s Product Management skills
- 4. Free Product Management Resources BOOKS EVENTS JOB PORTAL COMMUNITIES bit.ly/product_resources COURSES
- 6. A/B Testing in Plain English Nicole Lau September 2020
- 7. Agenda My Background What Is A/B Testing, and Why Should You A/B Test? Case Study - Concepts - Best Practices - Common Pitfalls Resources
- 9. Journey to eBay Product Manager, Experimentation Before eBay - Venture Capital Analyst-turned Program Manager - Business Development at startups and mid size companies At eBay - Experimentation Services to internal teams (i.e. E2E consultation) - Program Management (i.e. education, best practices)
- 10. What Is A/B Testing? Why Should You A/B test?
- 11. What is A/B testing? To see which variation performs better
- 12. Why Should You A/B test? Experimentation drives product innovations
- 14. Case Study
- 15. Scenario: You are eBay Homepage Product Manager
- 16. End-to-end Flow A sufficient backblog of quality testing ideas Hypothesis, success metrics, launch decision framework Configure, smoke test, ensure correct tracking in place Collect data Statistical significance and power Gather insights from both successes and failures
- 17. 1. Ideate Build sufficient backlog of quality ideas + Strategic Prioritization
- 18. 2. Design Hypothesis IF (improve headline wordings), THEN (users are more likely to click on CTA button). Success Metrics Primary- CTR on CTA button Secondary- Homepage bounce rate, module engagement Guardrail- Bought items, Revenue Launch decision framework
- 19. 3. Set up Correct user experience Correct tracking Free of data quality issues Version A (Control) Version B (variation)
- 20. 4. Run Reach minimum sample size needed No mid-test changes to avoid attribution issues https://unbounce.com/ab-test-duration-calculator/ Regression to the mean
- 21. 5. Analyze Ship, iterate, or abandon? 95% Statistical Significance 80% Statistical Power Confidence Level Impressions Clicks CTR Improvement p-value Min. sample reached? Version A (Control) 120,000 12,000 10% +5% (+/- 0.5) 0.03 Yes Version B (Variation) 120,000 18,000 15%
- 22. 6. Learn Learn from both successes and failures Uncover segment-specific insights and discoveries for future iterations of ideas
- 23. Resources
- 24. Resources Blogs - CXL (courses, webinars, and blogs on conversion optimization, ecommerce best practices, A/B testing guide) - VWO (“Knowledge Base” articles on A/B testing concepts) - Optimizely (“Knowledge Base” articles on A/B testing concepts) - Medium #experimentation Books - Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing by Ron Kohavi, Diane Tang, Ya Xu - Experimentation Works: The Surprising Power of Business Experiments by Stefan Thomke
- 25. My Contact Email: wylau.nicole@gmail.com LinkedIn: https://www.linkedin.com/in/nicole-lau-01414517/
- 26. www.productschool.com Part-time Product Management Training Courses and Corporate Training