How Experimentation Plays a Role in PM by Expedia Sr. PM
•Download as PPTX, PDF•
1 like•298 views
Learn more about testing, for example, what kind of product changes can be tested with standard experimentation techniques.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to How Experimentation Plays a Role in PM by Expedia Sr. PM
Similar to How Experimentation Plays a Role in PM by Expedia Sr. PM (20)
More from Product School
More from Product School (20)
Recently uploaded
Recently uploaded (20)
How Experimentation Plays a Role in PM by Expedia Sr. PM
- 1. www.productschool.com How Experimentation Plays a Role in PM by Expedia Sr. PM
- 2. FREE INVITE Join 30,000+ Product Managers on
- 3. COURSES Product Management Learn the skills you need to land a product manager job
- 4. COURSES Coding for Managers Build a website and gain the technical knowledge to lead software engineers
- 5. COURSES Data Analytics for Managers Learn the skills to understand web analytics, SQL and machine learning concepts
- 6. COURSES Digital Marketing for Managers Learn how to acquire more users and convert them into clients
- 8. A quick introduction Dane Molter Engineering + Technical PM, Trading Systems Product, Loyalty Product, Delivery Systems Product, Corporate Performance Analytics Product, Mobile
- 9. Disclaimers I’m a PM. This is coming from a PM’s perspective, and thus lacks some of the more nitty gritty technical detail you’d get from a data analyst. Moreover, this is largely 101 material, so if you are interested in more of a deep dive, there is a plethora of great material out there, so study away. There’s no such thing as too much awareness of the impacts experimentation can have on your product no matter what your level. Let’s go!
- 10. Remember this? Question TestHypothesizeResearch Analyze Conclude
- 11. Where are we experimenting? Jeff Patton, https://www.jpattonassociates.com/dual-track-development/
- 12. In both tracks! Discovery ● Lost cost, “smoke” or validation tests Develop ● High cost, “validated” experiments Note: the kinds of experiments we’re talking about here both cost engineering resources.
- 14. A/B/N Testing Concurrently changing a single variable, possibly multiple times 24-hour deals Save 40% or more on select hotels. Deals change over daily at midnight. 24-hour deals Save 40% or more on select hotels. Deals change over daily at midnight. Control 33% of traffic 24-hour deals Save 40% or more on select hotels. Deals change over daily at midnight. Variant 1 33% of traffic Variant 2 33% of traffic
- 15. Multivariate Testing Concurrently changing multiple variable, possibly changing each multiple times 24-hour deals Save 40% or more on select hotels. Deals change over daily at midnight. 24-hour deals Save 40% or more on select hotels. Deals change over daily at midnight. One day deals Save 40% or more on select hotels. Deals change over daily at midnight. One day deals Save 40% or more on select hotels. Deals change over daily at midnight. 24-hour deals Save 40% or more on select hotels. Deals change over daily at midnight. One day deals Save 40% or more on select hotels. Deals change over daily at midnight. Control 16% of traffic Variant 1 16% of traffic Variant 2 16% of traffic Variant 3 16% of traffic Variant 4 16% of traffic Variant 5 16% of traffic
- 16. Pre/Post Testing Consecutively changing a variable, sometimes multiple variables 24-hour deals Save 40% or more on select hotels. Deals change over daily at midnight. 24-hour deals Save 40% or more on select hotels. Deals change over daily at midnight. Variant 1 (Post) 100% of traffic later Control (Pre) 100% of traffic now
- 17. Setting up your tests
- 18. Define a hypothesis Question TestHypothesizeResearch Analyze Conclude
- 19. We should test removing phone number entry field on the Learn More page because someone in user testing told us they didn’t like it. Let’s test removing phone number entry field on the Learn More page. I hypothesize that by removing the phone number entry field from our Learn More page, users will be less impeded and will inquire more often, without impacting our ability to provide a response.
- 20. Understand the data you’ll need What are you collecting (or need to)? What are the primary/secondary metrics? Orders Conversion Rate Visitors New VisitorsRepeat RateErrorsClickthrough
- 21. Analyzing tests
- 24. Validate results with Test Significance Confidence Level Confidence Interval Sample Size States how many times your test result will be in a specific range out of 100 runs 90% Shows the range in which a test result lies (margin of error). We want this to be low +/- 1.6% The number of people in the test (incl. Control). Increasing this will decrease CI 569,887 visitors
- 25. Control Variant 1 % Difference CI Visitors 739,055 739,048 Purchasers 19,905 20,124 Conversion Rate 2.69% 2.72% 1.1% +/- 1.6% Confidence Level: 90%
- 27. Right Hand Left Hand % Difference CI Coin Flips 100 100 Heads 51 49 Heads Rate 51.0% 49.0% -3.9% +/- 22.8% Confidence Level: 90%
- 28. Right Hand Left Hand % Difference CI Coin Flips 1,000 1,000 Heads 510 490 Heads Rate 51.0% 49.0% -3.9% +/- 7.2% Confidence Level: 90%
- 29. Right Hand Left Hand % Difference CI Coin Flips 1,000,000 1,000,000 Heads 510,000 490,000 Heads Rate 51.0% 49.0% -3.9% +/- 0.2% Confidence Level: 90%
- 30. ConversionRate Days Running +1.0% +2.0% +3.0% -1.0% -2.0% -3.0% 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
- 31. Control 50% of traffic Variant 1 50% of traffic We hypothesized that by landing users on Travel, we could increase overall point usage, especially for the “point hoarders” segment of users, without negatively impacting the health of the overall business/other segments.
- 32. Control Variant 1 Visitors 154,988 155,160 Point Usage 308MM 312MM +1.29% Point usage uplift
- 33. Control Variant 1 Visitors 154,988 155,160 Point Usage 308MM 312MM Call Volume 15,498 23,274 +50% Relative call volume uplift +1.29% Point usage uplift
- 34. Control Variant 1 Visitors 154,988 155,160 Point Usage 308MM 312MM Call Volume 15,498 23,274 +50% Relative call volume uplift +1.29% Point usage uplift $ Annual savings $$$ Annual cost
- 36. “Isn’t this kind of testing just good for ecommerce?”
- 37. “My CEO told me to build it, so do I need to test it?”
- 38. “Can I run too many tests at one time on a single page?”
- 39. YES, tests can collide… ● and without much scale, this can create two possible problems: ○ False positives ○ Attribution of test to outcome BUT… ● if running multiple tests at scale, randomization and statistical significance should take care of most of the risk. Carefully consider your test setup (data points, format) and hypothesis!
- 40. Other common “gotchas” Even seasoned experimentation vets fall for these ● Testing during times where your users are acting fundamentally different than the norm ○ E.g. holidays ● Feature novelty ○ Sometimes, CTR != Definite good of your product ● Product loops / echo chambers ○ E.g. Testing pricing algorithms in a dual-sided marketplace ● Testing too many variations
- 41. Where you come in
- 42. Getting the ball rolling in your day-to-day Chances are, you or your company may be thinking along these lines already, so ask if there are any tools already in use to help!
- 43. Experimentation is a mindset for a PM
- 44. www.productschool.com Part-time Product Management, Coding, Data and Digital Marketing courses in San Francisco, Silicon Valley, New York, Santa Monica, Los Angeles, Austin, Boston, Boulder, Chicago, Denver, Orange County, Seattle, Bellevue, Toronto, London and Online
Editor's Notes
- Product management has quickly evolved from just this person who decides what to build into a role that not only decides what to build, but is completely on the hook for its success. You’ll need to know and action this scientific method in your every day.
- Before starting a test, clearly define an assumption about the outcome of your experiment. Remember, this is after the Question and Research phases of the Scientific Method.
- Autodispatch system Tuesday to Tuesday pre/post. We called this test, initially, in less than 2 hours. What did we do wrong?
- Calculation for your confidence interval. This translates to Sample Mean +/- (Z Score for your Confidence Level * (Standard Deviation / Sq Root of the Sample Size))
- Obviously, the higher the CL, the more likely the outcome you want. Similarly, the more you can do to drive down the CI, the better.
- If you flip the coin 100 times with each hand, you may achieve different results in the number of heads you get. But, this doesn’t mean that there really is a difference! Use the Significance of the test to reduce signal from noise. In this example, there’s still a 78% chance that this is due to chance!
- If you increase the number of flips, your CI tightens, and you can narrow down on the true significance of the test.
- Let’s keep going
- And sometimes, tests look like they might be winners, but are actually losers! Check out this great, well-worded hypothesis. Cool right?
- At first glance, all looks good.
- But, let’s dig a little deeper on secondary metrics. Oops.
- While very good for customer-facing ecomm experiences, this is about much more! Consider the optimization of your product’s performance and how you want to invest in dev later. Also, this is a key tool for learning at scale to validate your hypothesis and better understand your customer’s behaviors.
- Yes! You are on the hook for understanding your product, including all things that are impacting its performance, what makes your users tick, and what should be invested in in the future. Plus, wouldn’t you like to be able to push back on your CEO in the future?
- This one is a little more confusing
- Sometimes the tools in place aren’t always the best, though. Work with your engineering team to lay out what metrics you’re looking for and how you’ll be able to measure when tests are bucketed. There are lots of solutions out there that can be purchased, especially for ecommerce style tests, or you can try to wing it ad hoc. The important part is simply data collection. At GH, for example, we had to come up with a good tool to test algorithm impacts to our delivery system. Obviously, these did not work, so we started small. We focused on data collection up front, then ran everything as a pre/post to analyze impact in the real world. From there, we looked at creative ways to split traffic effectively.
- Learning is the absolutely critical takeaway here. Think the thing you are building is already a good idea? Test it anyway! The most you know about your product, the better you’ll make decisions about what is going to work and what’s worth your time. And remember, more tests don’t always mean better results. It just means more learning. Make this simply a tool in your toolbelt.