Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Scientific Method of Experimentation by Google PM


Published on

Product Management Event at #ProductCon Seattle on the Scientific Method of Experimentation by Ruben Lozano, Product Manager at Google Cloud.

Published in: Technology
  • Earn Up To $316/day! Easy Writing Jobs from the comfort of home! ♣♣♣
    Are you sure you want to  Yes  No
    Your message goes here
  • If you want to enjoy the Good Life: making money in the comfort of your own home with just your laptop, then this is for YOU... ♥♥♥
    Are you sure you want to  Yes  No
    Your message goes here
  • Stop Chasing Odds... Learn How To Master Them ▲▲▲
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

The Scientific Method of Experimentation by Google PM

  1. 1. The Scientific Method of Experimentation by Google PM
  2. 2. Join 35,000+Product Managers on Free Resources Discover great job opportunities Job Portal
  3. 3. C O U R S E S Product Management Learn the skills you need to land a Product Manager job
  4. 4. C O U R S E S Coding for Managers Build a website and gain the technical knowledge to lead software engineers
  5. 5. C O U R S E S Data Analytics for Managers Learn the skills to understand web analytics, SQL and machine learning concepts
  6. 6. C O U R S E S Learn how to acquire more users and convert them into clients Digital Marketing for Managers
  7. 7. C O U R S E S UX Design for Managers Gain a deeper understanding of your users and deliver an exceptional end-to- end experience
  8. 8. C O U R S E S For experienced Product Managers looking to gain strategic skills needed for top leadership roles Product Leadership
  9. 9. C O U R S E S Corporate Training Level up your team’s Product Management skills
  10. 10. Ruben Lozano T O N I G H T ’ S S P E A K E R
  11. 11. The scientific method of experimentation Seattle, WA | June 25, 2019
  12. 12. A/B
  13. 13. User Research 101 For Product Managers
  14. 14. What is user research? Systematic approach to discovering users' aspirations, goals, tasks, needs, pain points, and information and interaction requirements. User research grounds, verifies, and validates what a team builds.
  15. 15. Where does user research fit to product? Iterative research Foundational research Evaluative research PRODUCT DEVELOPMENT
  16. 16. Dimensions of user research methods Context Natural or near-natural Scripted Not using the product A hybrid of the above Attitudinal what people say Behavioral what people do 1 Qualitative answers why Quantitative answers how much/many 2 3 Source: Experiments
  17. 17. Experiments 101 What is an experiment?
  18. 18. What is an experiment? An experiment is a way to test a hypothesis about the product. An experiment may also refer to the gradual launch of a new feature. LIVE EVAL Note: Tests, while they are an important part of the software development journey, are not experiments, since you know in advance the result you expect
  19. 19. Why run live experiments?
  20. 20. I’m a PM. I know what will happen. Humans are terrible at making predictions 1. Hindsight bias 2. Observational selection bias 3. Projection bias 4. Anchoring bias … and hundreds of cognitive biases...
  21. 21. Doing a pre/post analysis is enough Brazil Search Traffic June 2014
  22. 22. A/B isolates the impact of just the product changes
  24. 24. Live experiments are not magic wands
  25. 25. “I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.‘” Abraham Maslow
  26. 26. bring the science
  27. 27. Fundamentals of experiment design The scientific method is an empirical method of acquiring knowledge. It is the systematic observation, measurement, and experimentation of a hypothesis. Observation1 Hypothesis2 Design3 Experiment4 Analysis5 Prove/Reject6
  28. 28. PM flavor of scientific method Observation1 Hypothesis2 Design3 Experiment4 Analysis5 Prove/Reject6 Ask a question0 Communicate results7
  29. 29. 0. Ask a question How can I increase usage of my product? How can I increase revenue attributed to my product? How can I increase user happiness? How can I simplify code without changing metrics? How can I affect click behavior?
  30. 30. 1. Observation: do background research What others have done before Are you doing something different? Did something change since the previous attempt? Quantitative data Behavioral metrics Surveys Trends Qualitative data Perceptions Attitudes Assumptions Preferences
  31. 31. 2. Develop a hypothesis A (1) testable (2) explanation for a phenomenon. The goal of an experiment is to prove or disprove the hypothesis. AVOID running experiments to see what happens or to gather data with no hypothesis. Use other user research methods and have a POV.
  32. 32. 2. Develop a hypothesis Example 1. Ask a question a. How can I increase sales for Prime users on the mobile app? 2. Do background research a. Users had troubles finding filters on mobile b. Users get overwhelmed with too many results c. Decreasing options simplifies decision-making d. BUT, past experiments limiting results had negative results
  33. 33. 2. Develop a hypothesis Hypothesis: Prime users will spend more $ if they can easily narrow their search results to prime products Is it valid? ● Is it testable? ● Does it have an explanation? ● Do I have an educated guess?
  34. 34. 3. Design experiment Hypothesis: Prime users will spend more $ if they can easily narrow their search results to prime products Design experiment: 1. Show a prime toggle on the navigation bar for all US prime users on the iOS app 2. Toggle off by default 3. No changes to a. Backend algorithms b. Logic that decides when to enable the prime filter c. Current prime filter behind the filter button
  35. 35. 3. Design experiment BTriggering criteria ● Who: US prime users using iOS app ● When: If results include a prime product ● How: Session-based Duration ● 2 weeks Launch criteria (success metric) ● Statistically significant increase in revenue ● No increase in latency
  36. 36. 4. Run experiment A B
  37. 37. 5. Analyze the data BResults +2.5% Revenue [1.9%, 3.1%], p=0.05
  38. 38. 5. Analyze the data 1. Statistical significance is the likelihood that the numeric difference between a control and treatment outcome is not due to random chance 2. Null hypothesis states there is no significant difference between control and treatment, any observed difference is due to sampling or experimental error 3. P-value evaluates how well the sample data supports the argument that the null hypothesis is true. A low p value suggests you can reject the null hypothesis 4. Confidence interval is a range of values (lower and upper bound) that is likely to contain an unknown population parameter
  39. 39. 5. Analyze the data significantly positive significantly negative inconclusive flat* (still inconclusive) (-) 0% (+) -0.5% | practical significance Results +2.5% Revenue [1.9%, 3.1%], p=0.05
  40. 40. 6. Draw conclusions Hypothesis: Prime users will spend more $ if they can easily narrow their search results to prime products 1. Validate data 2. Craft a story 3. Evaluate results a. Arguments in favor and against it b. Key observations and durable learning c. Next steps B
  41. 41. 7. Communicate results
  42. 42. “The most exciting phrase to hear in science, the one that heralds new discoveries, is not ‘Eureka‘ but ‘That’s funny…’” Isaac Asimov
  43. 43. Best practicesfrom my time at Amazon and at Google
  44. 44. Choose the right metrics 1. Think both short-term and long-term 2. Use metrics that matter 3. Align on the success metrics beyond your own team
  45. 45. Be a good wannabe scientist 1. The scientific method is not a suggestion 2. Be suspicious if you didn’t predict a specific result in advance 3. The more you slice and dice your data, the more false positives you’ll get 4. Lean against rolling out flat experiments, unless there are valid reasons
  46. 46. Create and follow templates and processes 1. Setup an intake process to get ideas from everyone 2. Establish a pre and post-experiment design template 3. Document all learnings and make them widely available
  47. 47. Thank you “Somewhere, something incredible is waiting to be known” Carl Sagan
  48. 48. Part-time Product Management, Coding, Data Analytics, Digital Marketing, UX Design, Product Leadership courses and Corporate Training