1.
Lukas Vermeer
Director of Experimentation @ Booking.com
Experimentation Island 2019
One neat trick
to run better
experiments

2.
Sample Ratio Mismatch
1. Why should we care?
2. How will we know?
3. Who suffers from SRM?
4. What can cause SRM?
5. What can we do?

3.
Why should
we care?
The pitfalls of
experimenting on the web

4.
12%
Lower average weight for
participants who described
“how applying versus not
applying eyeliner could make
them feel differently.”
Scientists in Chicago
have discovered a
revolutionary new
way to lose weight.
Read More >

5.
How will we
know?
Detecting sample ratio
mismatch

6.
Sample Ratio Mismatch
A mismatch between
the expected sample
ratio, and the observed
sample ratio.

7.
2
Number of commercially available
experimentation platforms (that I am
aware of) that are actively checking
for—and warning users about—SRM.

8.
Who suffers
from SRM?
Prevalence of SRM in
practice

9.
6%-10%
Percentage of experiments affected by SRM in companies
leading in experimentation which have been actively detecting
and addressing SRM issues for the past few years.
(Likely worse for companies which haven’t been checking at all.)

10.
What can
cause SRM?
A taxonomy and rules of
thumb for practitioners

11.
What can
we do?
One neat trick to run
better experiments

12.
One neat trick to run better
experiments
1. Check for SRM
2. ????????
3. More reliable results
4. ????????
5. Profit!

13.
Rules of thumb
1. Examine scorecards
2. Examine user segments
3. Examine time segments
4. Analyse performance metrics
5. Analyse engagement metrics
6. Count frequency of SRMs
7. Examine AA experiment
8. Examine severity
9. Examine downstream
10. Examine across pipelines

14.
Github Project
github.com/lukasvermeer/srm
Taxonomy Paper
dl.acm.org/citation.cfm?id=329
2500.3330722

15.
Lukas Vermeer
Director of Experimentation @ Booking.com
One neat trick
to run better
experiments

16.
Bonus slide for when someone inevitably asks “but what about Bayesian!?” 🙄