Controlled Experimentation toGuide Product Innovation!                !                Rajesh Parekh!
Controlled Experimentation (A/B Testing)!                                                •  Method to study effects of a t...
Why Run Controlled Experiments?!•  Commonly used approach in clinical trials! - What is the effect of a particular drug / ...
Controlled Experimentation: Use Cases! #     A"B"Stract"Widget"Company"                   A"B"Stract"Widget"Company"     _...
Controlled Experimentation: Use Cases! #           Free)Trial)                 Play)Now)              Mobile)Call)to)Ac=on...
Controlled Experimentation: Use Cases! #                Top)deal)highlighted)             Email)Template)Design)          ...
Controlled Experimentation: Use Cases! #     Backend)changes)(e.g.,)Personaliza=on)Algorithm))                      Contro...
Controlled Experimentation: Use Cases! #                           •  Follow-up message for users                         ...
Key Components of an Experimentation Platform!  Hashing function!                                   Metrics – suite of KPI...
Ensure Identical Control and Treatment!                                                                 Gender""•  Custom ...
A/A Tests!•  Run an experiment with two identical variants#•  Helps to determine if:# - Users are being split uniformly at...
Monitor Each Variant!•  Place yourself in each variant   to validate the experience#!•  Wrong sort order!!!!              ...
Objective Function!##     Conversion"                                                       Revenue"       P(conversion)) ...
Measure Overall Impact!•  Test focuses on# - A particular area of   the website# - A sub-population of   users#•  Measure!...
Panel Discussion: Questions!           Controlled)Experimenta=on)Panel)   15
Acknowledgements#Thanks to many talented individuals at Groupon I am privileged to work with!#•  Data Science#•  Engineeri...
Rajesh Parekh!                               Groupon!                               rajesh@groupon.com!                   ...
Upcoming SlideShare
Loading in...5
×

Groupon_Controlled Experimentation_Panel_The Hive

257

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
257
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Groupon_Controlled Experimentation_Panel_The Hive

  1. 1. Controlled Experimentation toGuide Product Innovation! ! Rajesh Parekh!
  2. 2. Controlled Experimentation (A/B Testing)! •  Method to study effects of a treatment
 # •  Concept:! - Randomly split users into two groups# Randomly) Divide)  A : Control#  B: Treatment# - A and B are identical to each other except A)(Control)) B)(Treatment)) for the treatment being evaluated# - Collect performance metrics from the experiment# - Run statistical tests to determine if differences between A and B are purely by chance# Measure)&) Evaluate) Controlled)Experimenta=on)Panel) 2
  3. 3. Why Run Controlled Experiments?!•  Commonly used approach in clinical trials! - What is the effect of a particular drug / treatment?#•  Systematically validate hypotheses with data!!•  Concurrently run the treatment and control! - The difference (if any) is#  Because of the treatment OR#  Due to random chance#•  Determine if a treatment is causal in nature! - E.g., Making the search box bigger causes increase in queries / user# Controlled)Experimenta=on)Panel) 3
  4. 4. Controlled Experimentation: Use Cases! # A"B"Stract"Widget"Company" A"B"Stract"Widget"Company" _________________) _________________) _________________) _________________) _________________) _________________) _________________) BUY)NOW) _________________) BUY"NOW" Website)Variants) Controlled)Experimenta=on)Panel) 4
  5. 5. Controlled Experimentation: Use Cases! # Free)Trial) Play)Now) Mobile)Call)to)Ac=on) Controlled)Experimenta=on)Panel) 5
  6. 6. Controlled Experimentation: Use Cases! # Top)deal)highlighted) Email)Template)Design) Controlled)Experimenta=on)Panel) 6
  7. 7. Controlled Experimentation: Use Cases! # Backend)changes)(e.g.,)Personaliza=on)Algorithm)) Controlled)Experimenta=on)Panel) 7
  8. 8. Controlled Experimentation: Use Cases! # •  Follow-up message for users who previously clicked on an ad# •  Incentive campaign to re- engage lapsed users# •  Think of this as placing filters / guards on a randomly chosen user population# Custom)Defined)User)Segments) Controlled)Experimenta=on)Panel) 8
  9. 9. Key Components of an Experimentation Platform! Hashing function! Metrics – suite of KPI! Group)0) ! ! Revenue" ! ! F()))))))))))))) Group)1) ! ! Time"Spent" Abandonment" Click>Through"Rate" ! Group)NU1) ! ! ! Session"Length" Purchase"Rate" Logging! Dashboard! ! ! ! ! •  Detailed)logging)of)all)user)interac=ons) •  Metric)improvements)and)Sta=s=cal) Significance)in)a)central)place) Controlled)Experimenta=on)Panel) 9
  10. 10. Ensure Identical Control and Treatment! Gender""•  Custom Segments# Male) Female)•  Frequency Distribution# CONTROL" TREAMENT" Region"Size" Small) Medium) Large) CONROL" TREATMENT" Prior"Exposure"•  Large Difference in Prior Exposure Rate violates δ%" assumptions# No) Yes) CONROL" TREATMENT" Controlled)Experimenta=on)Panel) 10
  11. 11. A/A Tests!•  Run an experiment with two identical variants#•  Helps to determine if:# - Users are being split uniformly at random# - Correct data is being logged# - Variance between identical populations of users is acceptable#•  Challenge:! - Few purchases of high value deals render statistically significant difference between treatment and control# SPAIN"TRIP" $1,999" Controlled)Experimenta=on)Panel) 11
  12. 12. Monitor Each Variant!•  Place yourself in each variant to validate the experience#!•  Wrong sort order!!!! Carefully)inspect)each)variant) Controlled)Experimenta=on)Panel) 12
  13. 13. Objective Function!## Conversion" Revenue" P(conversion)) E(rev))=)P(conversion))*)price) ) ) •  Favors)lower) •  More)expensive)deals)can) price)deals) dominate) Need)to)balance)mul=ple,)oZen)conflic=ng)objec=ves)) Controlled)Experimenta=on)Panel) 13
  14. 14. Measure Overall Impact!•  Test focuses on# - A particular area of the website# - A sub-population of users#•  Measure! - Improvement on the sub-segment AND# - Entire population!# Measure)overall)impact)to)guard)against)cannibaliza=on) Controlled)Experimenta=on)Panel) 14
  15. 15. Panel Discussion: Questions! Controlled)Experimenta=on)Panel) 15
  16. 16. Acknowledgements#Thanks to many talented individuals at Groupon I am privileged to work with!#•  Data Science#•  Engineering#•  Marketing / Market Research# Controlled)Experimenta=on)Panel) 16
  17. 17. Rajesh Parekh! Groupon! rajesh@groupon.com! !Controlled)Experimenta=on)Panel) 17
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×