The document discusses controlled experimentation (A/B testing) as a method to study the effects of treatments on users. It notes that experiments randomly divide users into a control and treatment group, with the only difference being the treatment evaluated. Performance metrics are collected and statistically analyzed to determine if any differences are due to the treatment or random chance. Examples of experiments include variations to website design, mobile calls to action, and personalization algorithms. Key aspects of experimentation platforms include hashing to randomly assign users, detailed logging, metrics dashboards, and ensuring control and treatment groups are identical. The document emphasizes measuring overall impact beyond just segments under treatment.

1. Controlled Experimentation to
Guide Product Innovation!
!
Rajesh Parekh!

2. Controlled Experimentation (A/B Testing)!
• Method to study effects of a treatment
#
• Concept:!
- Randomly split users into two groups#
Randomly
Divide
➥ A : Control#
➥ B: Treatment#
- A and B are identical to each other except
A
(Control)
B
(Treatment)
for the treatment being evaluated#
- Collect performance metrics from the
experiment#
- Run statistical tests to determine if
differences between A and B are purely
by chance#
Measure
&
Evaluate
Controlled
Experimenta=on
Panel
2

3. Why Run Controlled Experiments?!
• Commonly used approach in clinical trials!
- What is the effect of a particular drug / treatment?#
• Systematically validate hypotheses with data!
!
• Concurrently run the treatment and control!
- The difference (if any) is#
➥ Because of the treatment OR#
➥ Due to random chance#
• Determine if a treatment is causal in nature!
- E.g., Making the search box bigger causes increase in queries / user#
Controlled
Experimenta=on
Panel
3

4. Controlled Experimentation: Use Cases!
#
A
B
Stract
Widget
Company
A
B
Stract
Widget
Company
_________________
_________________
_________________
_________________
_________________
_________________
_________________
BUY
NOW
_________________
BUY
NOW
Website
Variants
Controlled
Experimenta=on
Panel
4

5. Controlled Experimentation: Use Cases!
#
Free
Trial
Play
Now
Mobile
Call
to
Ac=on
Controlled
Experimenta=on
Panel
5

6. Controlled Experimentation: Use Cases!
#
Top
deal
highlighted
Email
Template
Design
Controlled
Experimenta=on
Panel
6

7. Controlled Experimentation: Use Cases!
#
Backend
changes
(e.g.,
Personaliza=on
Algorithm)
Controlled
Experimenta=on
Panel
7

8. Controlled Experimentation: Use Cases!
# • Follow-up message for users
who previously clicked on an
ad#
• Incentive campaign to re-
engage lapsed users#
• Think of this as placing filters /
guards on a randomly chosen
user population#
Custom
Defined
User
Segments
Controlled
Experimenta=on
Panel
8

9. Key Components of an Experimentation Platform!
Hashing function! Metrics – suite of KPI!
Group
0
! ! Revenue
!
!
F(
)
Group
1
!
!
Time
Spent
Abandonment
Click-­‐Through
Rate
! Group
N-­‐1
!
! ! Session
Length
Purchase
Rate
Logging! Dashboard!
!
!
!
!
• Detailed
logging
of
all
user
interac=ons
• Metric
improvements
and
Sta=s=cal
Significance
in
a
central
place
Controlled
Experimenta=on
Panel
9

10. Ensure Identical Control and Treatment!
Gender
• Custom Segments#
Male
Female
• Frequency Distribution#
CONTROL
TREAMENT
Region
Size
Small
Medium
Large
CONROL
TREATMENT
Prior
Exposure
• Large Difference in Prior
Exposure Rate violates δ%
assumptions# No
Yes
CONROL
TREATMENT
Controlled
Experimenta=on
Panel
10

11. A/A Tests!
• Run an experiment with two identical variants#
• Helps to determine if:#
- Users are being split uniformly at random#
- Correct data is being logged#
- Variance between identical populations of users is acceptable#
• Challenge:!
- Few purchases of high value deals render statistically significant
difference between treatment and control#
SPAIN
TRIP
$1,999
Controlled
Experimenta=on
Panel
11

12. Monitor Each Variant!
• Place yourself in each variant
to validate the experience#
!
• Wrong sort order!!
!
!
Carefully
inspect
each
variant
Controlled
Experimenta=on
Panel
12

13. Objective Function!
#
#
Conversion
Revenue
P(conversion)
E(rev)
=
P(conversion)
*
price
• Favors
lower
• More
expensive
deals
can
price
deals
dominate
Need
to
balance
mul=ple,
oZen
conflic=ng
objec=ves
Controlled
Experimenta=on
Panel
13

14. Measure Overall Impact!
• Test focuses on#
- A particular area of
the website#
- A sub-population of
users#
• Measure!
- Improvement on the
sub-segment AND#
- Entire population!#
Measure
overall
impact
to
guard
against
cannibaliza=on
Controlled
Experimenta=on
Panel
14

15. Panel Discussion: Questions!
Controlled
Experimenta=on
Panel
15

16. Acknowledgements#
Thanks to many talented individuals at Groupon I am privileged to work with!#
• Data Science#
• Engineering#
• Marketing / Market Research#
Controlled
Experimenta=on
Panel
16

17. Rajesh Parekh!
Groupon!
rajesh@groupon.com!
!
Controlled
Experimenta=on
Panel
17