Experiments taken to a broader level

1. Next Speaker:
Chad
Sanderson (USA)
Experiments taken
to a broader level
Next Speaker:

2. Hello!
I am Chad Sanderson
Product Manager, amateur data scientist, dog father, and
conversion optimization practitioner. I like getting smart
about testing.
5
@chadjsanderson

3. Beyond Traditional A/B Testing

4. What is a Digital Randomized Controlled Trial?

5. What is the value of Digital RCT’s?
Automated Randomization
Treatment Assignment, or the method of distributing users into bucketed groups
Experimental Statistics
Methods to calculate aggregate metrics and conduct ‘tests’ using measured
variance. Frequentist techniques offer strong error quantification
Boundaries & Safeguards
Quickly determine whether or not new features are negatively impacting
performance or user metrics

6. Well…not quite
SRM
Representation
Seasonality
Cookie Churn

8. “Experimentation [CRO] is a framework for
quantifying the risk of a decision by inferring causality based on data.”

9. Areas of Business Risk
• Hiring a new team member
• Executive leadership performance
• Purchase of new tools
• Launching loyalty programs
• Changing a creative direction
• Beginning a social media campaign
• Evaluating a vendor
• Writing documentation guidelines
• Making a sales pitch
• Responding to a customer

10. ‘Strict Randomized Controlled Trials’
All hail our robot overlords
Time Series Analysis
Pre-Post? No problem
Factorials / MVT’s
Finding the right levers to pull
‘Loose Randomized Controlled Trials’
Getting our hands dirty

12. Small Sample Testing
A convenience sample of 789 adults (248 women and 541 men) was recruited
for the study during hospital admission

14. Remember what matters
Representation! Sample size DOES NOT affect the accuracy of
your data so long as what you collect is representative of
your underlying population.

15. Garbage in, Garbage out
-18%--
Product Teams who pre-registered test designs experienced a 18%
reduction in submission volume but the % of winning experiments
remained fixed.

16. Segmented
Regression
• Randomization isn’t always
possible
• Regression can be used to
identify correlations in data
• ‘Time before and after a
change point’ can be used as
an independent variable
• Historical data can be used
• Control is not needed!

17. Factorial Experiments
• In factorial experiments, we typically are
working with very small sample sizes, where
each sample is potentially expensive
• What we care about is not the BEST
performing combinations of treatments, but
whether there are sizeable main effects or
interaction effects that can inform us which
‘levers to pull.’
• Continual optimization can be performed by
‘hill climbing’ without the need of large
sample sizes, hypotheses, or even p values.

18. Getting more $$$ for your $$$
The social media team was small (1
person) and wanted to understand
how to optimize advertising spend.
Because each experiment was high
cost, a factorial experiment was the
perfect tool to understand important
factors:
1. Advertisement location
a. Newsfeed
b. Sidebar
2. Audience Age
a. Under 25
b. Over 45

19. Holy interaction plots, Batman!
Even with relatively few samples, we were able to determine that the group Age made a dramatic
difference in ad performance, while there also appeared to be a weak interaction between Age and
Advertisement Location. All age groups had better performance when ads were targeted to the
Newsfeed, but young people seemed to respond especially poorly for the right hand panel ads.

21. Rules of the Road
1. Understand your limits. Take it slow!
2. Take the data for what it is. Accept imperfection. Data doesn’t always tell the full
story, and that’s okay.
3. Experiment for ROI. A cheap test that could help avert a major disaster? Great!
An expensive test that won’t tell you much of anything? Not great.
4. And uh…don’t do anything that would get you arrested.

22. Good luck!