1.
www.productschool.com
How to Correctly Use
Experimentation in PM by Google PM

2.
Join 35,000+Product
Managers on
Free Resources
Discover great job
opportunities
Job Portal
prdct.school/PSJobPortalprdct.school/events-slack

3.
C O U R S E S
Product
Management
Learn the skills you need to land
a Product Manager job

4.
C O U R S E S
Coding
for Managers
Build a website and gain the
technical knowledge to lead
software engineers

5.
C O U R S E S
Data Analytics
for Managers
Learn the skills to understand web
analytics, SQL and machine learning
concepts

6.
C O U R S E S
Learn how to acquire more users
and convert them into clients
Digital Marketing
for Managers

7.
C O U R S E S
UX Design
for Managers
Gain a deeper understanding of your
users and deliver an exceptional end-to-
end experience

8.
C O U R S E S
For experienced Product Managers
looking to gain strategic skills needed
for top leadership roles
Product
Leadership

9.
C O U R S E S
Corporate
Training
Level up your team’s Product
Management skills

10.
Ruben Lozano
T O N I G H T ’ S S P E A K E R

11.
A/B

12.
A B
CONTROL VARIATION
BUY NOW BUY NOW
21%
35%

13.
“I suppose it is tempting, if the only
tool you have is a hammer, to treat
everything as if it were a nail.‘”
Abraham Maslow

14.
Agenda
1. User Research 101
2. Experiments 101
a. Why run live experiments?
3. Deep Dive on Live Experiments
a. Process overview
i. MVT
ii. MAB
b. Common mistakes
c. Best practices
4. Questions

15.
“The most exciting phrase to hear
in science, the one that heralds
new discoveries, is not ‘Eureka‘ but
‘That’s funny...‘”
Isaac Asimov

16.
User Research
101
Love of learning

17.
What is user research?
User research is a systematic approach to discovering users' aspirations, goals,
tasks, needs, pain points, and information and interaction requirements.
Research grounds, verifies, and validates what a team builds.

18.
Types of user research
3-dimensional framework
1. Attitudinal vs. Behavioral
a. "what people say" vs "what people do"
2. Qualitative vs. Quantitative
a. qualitative methods are better suited for answering why, whereas quantitative methods do a
better job answering how many and how much
3. Context of Use
a. Natural or near-natural
b. Scripted
c. Not using the product
d. A hybrid Source:
https://www.nngroup.com/articl
es/which-ux-research-methods/

19.
Landscape of user research methods
Source:
https://www.nngroup.com/articl
es/which-ux-research-methods/
Key for context of
product use during
data collection
LIVE

20.
Where does research fit to product?
Iterative research
Foundational research
Evaluative research
PRODUCT DEVELOPMENT

21.
Foundational research
1. Purpose
a. Uncover opportunities
b. Inform product direction and strategy
c. Continuously engage with users
2. Research methods
a. Ethnographic field studies
b. Diary or camera studies
c. Competitive analysis
3. Study participants
a. Future and current users

22.
Iterative research
1. Purpose
a. Explore design concepts
b. Identify usability issues
2. Research methods
a. In-lab usability studies
b. Walkthroughs
c. User testing
3. Study participants
a. Future and current users, early access

23.
Evaluative research
1. Purpose
a. Test how the product performs with target users
b. Improve product
2. Research methods
a. Surveys
b. Lab usability studies
c. Live experiments
3. Study participants
a. Current or potential users

24.
Experiments 101
What is an experiment?

25.
What is an experiment?
An experiment is a way to test a hypothesis about the
quality of a product.
An experiment may also refer to the gradual launch of a
new feature.
Note: Tests, while they are an important part of the software
development journey, are not experiments, since you know
in advance the result you expect
LIVE
EVAL

26.
How does it work?
Total Population (e.g. users, requests, queries)
Unaffected
population
Control group
Treatment
group
Experiment Diversion
Default
Experimental
Value
Default

27.
Success metric
A quantifiable measure of a desirable outcome
Product Sample Metric
Facebook DAU (daily active users)
Airbnb Nights booked
WhatsApp Messages sent
Quora Number of questions answered
Amazon Revenue
Long-term
growth

28.
Why run live
experiments?

29.
Obviously, this is going to work
Humans are terrible at
making predictions
1. Hindsight bias
2. Identifiable victim effect
3. Observational selection bias
4. Projection bias
5. Anchoring bias
… and hundreds of cognitive biases...

30.
Just do a pre/post
The world is a fascinating
place.

31.
Just do a pre/post
Brazil Search Traffic
June 2014

32.
A/B isolates the impact of
just the product changes

33.
Is something a good idea?
A B
ORGANIC HONEYCRISP APPLE
GRANNY SMITH APPLE
FUJI APPLE
ORGANIC HONEYCRISP APPLE
GRANNY SMITH APPLE

34.
What’s the best iteration of a good idea?
A
B
C
Yahoo! - Wikipedia, the free encyclopedia
en.wikipedia.org...
Yahoo! - Wikipedia, the free encyclopedia
en.wikipedia.org/wiki/Yahoo
Yahoo! - Wikipedia, the free encyclopedia
en.wikipedia.org/wiki/Yahoo

35.
Why do live experiments?
To learn
1. Is something a good idea?
2. What’s the best iteration of a good idea?
3. How much is the impact of something?
4. Is something causing a problem?
5. Is something still a good idea?
6. What is the long-term impact of something?

36.
Live
experiments
are not
magic wands

37.
bring the science

38.
Fundamentals of experiment design
The scientific method is an empirical method of acquiring knowledge. It is the
systematic observation, measurement, and experimentation of a hypothesis.
Observation1 Hypothesis2 Design3
Experiment4 Analysis5 Prove/Reject6

39.
Deep Dive on Live
Experiments
How to succeed?

40.
Process overview

41.
PM flavor of scientific method
1. Ask a question
2. Do background research
3. Develop a hypothesis
4. Test the hypothesis
5. Analyze data
6. Draw conclusions
7. Communicate results

42.
1. Ask a question
● How can I increase user happiness?
● How can I increase usage of my product?
● How can I simplify code without changing metrics?
● How can I increase revenue attributed to my product?
● How can I affect click behavior?

43.
2. Do background research
● What others have done before
○ Are you doing something different?
○ Did something change since the previous attempt?
● Quantitative data
○ Behavioral metrics
○ Funnel metrics
○ Downstream metrics
○ Surveys
○ Trends
● Qualitative data
○ Perceptions
○ Attitudes
○ Assumptions
○ Preferences

44.
3. Develop hypothesis
A testable explanation for a phenomenon
Before performing an experiment and observe what happens, we first clearly
identify what we "think" will happen. We make an "educated guess."
We set out to prove or disprove the hypothesis.

45.
3. Develop hypothesis
BUY NOW BUY NOW
It will make
the product
awesome!
In our focus
group, people
liked this
design better
Click-through
rate will
increase
The color red
is out of
harmony with
the world
Users
associate red
with danger

46.
3. Develop hypothesis
BUY NOW BUY NOW
Click-through
rate will
increase
Users
associate red
with danger

47.
Example
Hypothesis:
Prime users will spend more $ if they can easily narrow their
search results to prime products
Design experiment:
Show a prime toggle on the navigation bar for all US prime users
on iOS

48.
4. Test the hypothesis
A B

49.
4. Test the hypothesis
B1. Success metric
2. Triggering criteria
3. Duration of the experiment
4. Launch criteria
Option 1
+2.5% Revenue [1.9%, 3.1%], p=0.02
Option 2
+1.3% Revenue [-0.3%, 2.9%], p=0.15

50.
5. Analyze the data
Statistical significance is the likelihood that the numeric difference between a
control and treatment outcome is not due to random chance
Null hypothesis states there is no significant difference between control and
treatment, any observed difference is due to sampling or experimental error
P-value evaluates how well the sample data supports the argument that the null
hypothesis is true. A low p value suggests you can reject the null hypothesis
Confidence interval is a range of values (lower and upper bound) that is likely to
contain an unknown population parameter

51.
5. Analyze the data

52.
5. Analyze the data
significantly positive
significantly negative
inconclusive
flat* (still inconclusive)
(-) 0% (+)
-0.5% | practical significance

53.
6. Draw conclusions
Hypothesis:
Prime users will spend more $ if they can easily narrow
their search results to prime products
1. Validation
2. Description
3. Evaluation
a. Arguments in favor
b. Arguments against
c. Key observations
d. Durable learning
e. Next steps
B

54.
7. Communicate results

55.
Multivariate
Testing
Tests multiple variants
simultaneously to determine
how the combinations of
multiple variables influence the
output.

56.
APPLE
BUY NOW
BUY PURCHASE
APPLE
APPLE
APPLE
Font Color (4)
- Black
- Red
- Gray
- Blue
Button Label (3)
- BUY NOW
- BUY
- PURCHASE
Image (4)
- Option A
- Option B
- Option C
- Option D
48

57.
4. MVT
Pro
● Measures interactions between
elements
● Test multiple variants of specified
variables are tested in a user interface
Cons
● Much more traffic is typically required
● All combinations of the variants must
make sense together

58.
Multi-armed
Bandits
Optimizes the objective function
by updating the randomization
distribution as the experiment
progresses

59.
Comparing A/B/n vs MAB
A
B
C
week 1 2 3 4 5 6
A
B
C
week 1 2 3 4 5 6

60.
Multi-armed Bandits
Pro
● Produces answers far more quickly
● Safer to allocate larger fraction of traffic to
experiments
● Increases experimentation outputs
Cons
● Over valuing short-term behavior
● Lead to self-fulling properties where an initial
positive change is overly valued
● Can’t quantify impact from control

61.
Common mistakes

62.
Ethical
● Are you reinforcing biases?
● Are you using dark patterns to improve KPIs?
● Are you deceiving users?
● Are you violating privacy?
● What if the description of your experiment was
published on the NYT?

63.
Time-related
● Stopping tests too early
● Not testing long enough, or not waiting long enough
for the right metrics
● Testing for too long
● Tests are not run for full weeks
● Running a split test during the holidays
● Forgetting learnability

64.
Configuration
● Loose triggering
● Ignoring different traffic sources
● Sending too little traffic to variations
● Change experiment settings in the
middle of a test
● Not considering mobile traffic variables

65.
Design
● Tests are not based on a hypothesis
● Using too many metrics
● Using the metric that you can measure, not
the one that matters
● Not setting quantifiable goals
● Using too many changes on a single
treatment
● Using too many variations

66.
Analysis
● Trusting what you read from the numbers
● Ignoring the difference between local and
global metrics
● Not taking seasonal changes into account
● Ignoring technical issues (flickering,
latency, etc.)
● Incorrect post-test segmentation
● P-hacking or data fishing!

67.
Best practices

68.
Choose the right metrics
● Ask “How do we define success?” and “How do we measure user value?”
○ Think both short-term and long-term
○ Be more inclusive
● Should reflect changes that your product controls
● Avoid using lagging indicators
● Align on the success metrics beyond your own team

69.
Be a good wannabe scientist
● Bias against rolling out flat experiments, unless it’s a complete redesign
● Be suspicious if you didn’t predict a specific result in advance, retest to
verify
● Look for a consistent story in the metrics
● The more you slice and dice your data, the more false positives you’ll get
● Run an A/A test!

70.
Create and follow templates and processes
● Intake forms
● Pre-experiment design
● Post-experiment analysis
● Consolidation of results

71.
Pre-experiment design
● Experiment Information
○ Summary
● What’s changing?
○ Control vs treatment
● Objective
○ Goal and hypothesis
● Triggering criteria
○ Device, geography, target user, etc
● Launch criteria
○ Launch if..
○ Definition of flat, bonferroni correction
● Experiment duration

72.
Post-experiment analysis
● Executive Summary
○ Overview with final decision
● Reasons to Launch
○ Based on launch criteria
● Reasons not to Launch
○ Based on launch criteria
● Latency Impact
● Key observations
○ “That’s funny”
● Durable learnings
○ Overall insights
● Detailed analysis

73.
Implement an experimentation process
1. Intake request
2. Pre-experiment review
3. Monitoring experiment
4. Post-experiment review
5. Experimentation council
6. Socialize results
7. Document results

74.
www.productschool.com
Part-time Product Management, Coding, Data Analytics, Digital
Marketing, UX Design, Product Leadership courses and
Corporate Training