Download as PDF, PPTX
Uploaded byGrow.co
MAU Vegas 2016 — Lessons Learned From A/B Testing Wins and Losses
The document outlines lessons learned from A/B testing as shared by growth hacker Andy. It emphasizes the importance of building a testing engine pre and post product/market fit, creating hypotheses, designing effective tests, and continuously evaluating results for scalable growth. The document also highlights different types of tests and strategies for optimizing user engagement and product features.
More Related Content
Similar to MAU Vegas 2016 — Lessons Learned From A/B Testing Wins and Losses
MAU Vegas 2016 — Lessons Learned From A/B Testing Wins and Losses
- 1. Lessons Learned FromA/B Testing Wins and Losses
- 2. WHO I AM:GROWTH HACKER ANDY
- 3. Growth Team /groTH tem/n. A group within the company that develops the product experience in a way that creates more activated users ! Pre Product/Market fit: build the growth engine ! Post Product/Market fit: optimize the engine (make it bigger) ! Any time: generate hacks, non-obvious ways to reach your goals Definition
- 4.
- 5. GET THE PIECESIN PLACE BEFORE ADDING GAS
- 6. Do things that don’tscale
- 8. Test yourself toproduct market fit What type of product are our users interested in? How can we reach this users the best way? Pour Gas on it WE HAD TO EXPERIMENT WITH A LOT OF THINGS
- 9. How do youbuild what users actually want? Child ID Dog TrackerFamily Locator Lost Phone Ads Landing Pages
- 10. THINK OF ITAS DIVIDE & CONQUER
- 11.
- 13. RISKLOW HIGH Big Swings Optimizations REWARD HIGH StartUps need step function take bold bets
- 14. Scalable growth throughtesting Lather, Rinse, Repeat. Continuous Improvement Evaluate Results Test Design Develop Hypothesis Implement Leanings
- 15. Developing hypotheses How toDevelop Hypotheses 1. Leverage the team, institutional knowledge, understand important “Themes” 2. Qualitative studies, surveys 3. Quantitative analysis
- 16. Test Design Test Design How toDesign Tests Effectively 1. Determine right testing format 2. Optimize test to avoid overinvesting 3. Execute quickly, full instrumentation for analysis
- 17. Evaluate Results How toEvaluate Results 1. Analyze core metrics impact of each variant 2. Ensure all numbers are statistically significant 3. Understand long term impacts and side effects Evaluate Results
- 18. Implement Learnings How toImplement Learnings 1. Understand what caused the change 2. Determine how this learnings applies to product aspects 3. Promote visibility of learnings/takeaways Implement Leanings
- 19. Qualitative Tests PaintedDoor Tests Feature Tests INVESTMENT COSTLOW HIGH Painted Door Top of Funnel Test Full FeatureMany Some Few Types of tests avoid overinvesting
- 20. Qualitative Tests Purpose Determine interestin product via conversations with your users and potential users Key Test as many ideas as possible but keep in mind that people want to please you Example Emergency Contacts ETA Sharing Carpooling Qualitative Tests
- 21. Feature Tests Purpose Building afunctional feature and see how users interact Key Don’t build the full feature, start at the top of the funnel and work your way down Example Emergency Contacts Recommender FUE Add EC Send EC Accept EC Create ACT (Partial) Feature Tests
- 22. Aggressive vs. PassiveTests Passive Tests End Point Aggressive Tests LEARNINGSLOW HIGH
- 23. HOW TO TRUST YOURRESULTS?
- 24. Test Design -Explicit Control Variant Allocation Design variants to have an additional control group ➔ Ensures control group meets test user criteria ➔ Reduce risk of variant assignment issues
- 25. Test Design -Explicit Control Variant Allocation All Users
- 26. Test Design -Explicit Control Variant Allocation Eligible for experiment Ineligible for experiment
- 27. Test Design -Explicit Control Variant Allocation Eligible but excluded from experiment Ineligible for experiment Included in experiment
- 28. Test Design -Explicit Control Variant Allocation Treatment 1 Eligible but excluded from experiment Ineligible for experiment Control Variant 1 Variant 2
- 29. Test Design -Explicit Control Variant Allocation Always consider keeping long-term holdout groups ➔ Understand potential impacts that may not be seen early in the user lifecycle ➔ Enables measurement of how impacts rollup into product growth
- 30. “A GOOD PLAN, VIOLENTLY EXECUTEDTODAY, IS BETTER THAN A PERFECT PLAN EXECUTED NEXT WEEK.” GENERAL GEORGE S PATTON
- 31. HERE IS WHATI AM UP TO NEXT Andreas@reddit.com