Main Takeaways: - Test Lifecycle: From Ideation to production and post-analysis - Experiments & Features: Benefits of testing large features - Top Challenges of A/B Testing: Main challenges product managers face as it pertains to testing

1. JM Coaching & Training © 2020
www.productschool.com
A/B Testing Best Practices
by Hulu Product Leader

2. CERTIFICATES
Your Product Management Certificate Path
Product Leadership
Certificate™
Full Stack Product
Management Certificate™
Product Management
Certificate™
20 HOURS
40 HOURS
40 HOURS

3. Corporate
Training
Level up your team’s Product
Management skills

4. Free Product Management Resources
BOOKS
EVENTS
JOB PORTAL
COMMUNITIES
bit.ly/product_resources
COURSES

5. The business intelligence and analytics platform
for every need
www.gooddata.com
Sponsored by:

6. www.productschool.com
Webinar: A/B Testing Basic Principles
Amna Aftab, Hulu Product Manager

7. Hello, I’m Amna
S P E A K E R: A LITTLE BIT ABOUT ME
HIGHLIGHTS
Product Management for 5 Years & Hulu for 2 years
Own user experiences that drive subscriber growth across web & OTT
Responsible for launching new features, & leading growth experimentation
program
Put the user first
MY TENANTS AS A PM
Build scalable products that will make the next feature easier, and
faster to build
Work as one team

8. 1
1
Experimentation Lifecycle
Experiments & Features
Top A/B Testing Challenges
AGENDA
A/B TESTING BASIC PRINCIPLES
Types of Experiments

9. 1
1
Experimentation Life Cycle
Experiments & Features
Top Challenges of A/B Testing
AGENDA
A/B TESTING BASIC PRINCIPLES
Types of Experiments

10. EXPERIMENTATION LIFECYCLE
IDEATION
PRIORITIZE
REQS & DESIGN
DEV & QA
LAUNCH
ANALYSIS & LEARN
Identify problem and or/areas of opportunity & draft hypothesis. These ideas
can come from all sectors of the business, from the customer success to
engineering.
Use a prioritization method to rank against your experimentation roadmap.
Reqs are defined, UX executes on requirements & creates a design solution
Engineering builds & tests the treatments for the experiment
Experiment launches into production
Analyze outcome of the experiment and match back to the original hypothesis
NEXT STEPS Determine whether the treatment should be rolled out to production

11. 1
1
Experimentation Life Cycle
Experiments & Features
Top Challenges of A/B Testing
AGENDA
A/B TESTING BASIC PRINCIPLES
Types of Experiments

12. TYPES OF EXPERIMENTS
Discovery
Validation Evaluation
Optimization
LONG TERM
SHORT TERM

13. 1
1
Experimentation Life Cycle
Experiments & Features
Top Challenges of A/B Testing
AGENDA
A/B TESTING BASIC PRINCIPLES
Types of Experiments

14. “Why would we test something that we
want to roll into production anyway?”

15. TOP 3 REASONS TO TEST FEATURES
1. BUILD FOR SCALE 2. OBTAIN
LEARNINGS
3. PRODUCE HIGHER ROI

16. 1
1
Experimentation Life Cycle
Experiments & Features
Top Challenges of A/B Testing
AGENDA
A/B TESTING BASIC PRINCIPLES
Types of Experiments

17. TOP A/B TESTING CHALLENGES
1. DATA POLLUTION 2. TESTING VARIOUS VARIABLES 3. AVOIDING ITERATIVE TESTING

18. Thank You
Stay in Touch
www.linkedin.com/in/amnaaftab
www.productschool.com

19. www.productschool.com
Part-time Product Management Training Courses
and
Corporate Training