Your SlideShare is downloading. ×
Groupon_Controlled Experimentation_Panel_The Hive
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Groupon_Controlled Experimentation_Panel_The Hive

188
views

Published on


0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
188
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Controlled Experimentation toGuide Product Innovation! ! Rajesh Parekh!
  • 2. Controlled Experimentation (A/B Testing)! •  Method to study effects of a treatment
 # •  Concept:! - Randomly split users into two groups# Randomly) Divide)  A : Control#  B: Treatment# - A and B are identical to each other except A)(Control)) B)(Treatment)) for the treatment being evaluated# - Collect performance metrics from the experiment# - Run statistical tests to determine if differences between A and B are purely by chance# Measure)&) Evaluate) Controlled)Experimenta=on)Panel) 2
  • 3. Why Run Controlled Experiments?!•  Commonly used approach in clinical trials! - What is the effect of a particular drug / treatment?#•  Systematically validate hypotheses with data!!•  Concurrently run the treatment and control! - The difference (if any) is#  Because of the treatment OR#  Due to random chance#•  Determine if a treatment is causal in nature! - E.g., Making the search box bigger causes increase in queries / user# Controlled)Experimenta=on)Panel) 3
  • 4. Controlled Experimentation: Use Cases! # A"B"Stract"Widget"Company" A"B"Stract"Widget"Company" _________________) _________________) _________________) _________________) _________________) _________________) _________________) BUY)NOW) _________________) BUY"NOW" Website)Variants) Controlled)Experimenta=on)Panel) 4
  • 5. Controlled Experimentation: Use Cases! # Free)Trial) Play)Now) Mobile)Call)to)Ac=on) Controlled)Experimenta=on)Panel) 5
  • 6. Controlled Experimentation: Use Cases! # Top)deal)highlighted) Email)Template)Design) Controlled)Experimenta=on)Panel) 6
  • 7. Controlled Experimentation: Use Cases! # Backend)changes)(e.g.,)Personaliza=on)Algorithm)) Controlled)Experimenta=on)Panel) 7
  • 8. Controlled Experimentation: Use Cases! # •  Follow-up message for users who previously clicked on an ad# •  Incentive campaign to re- engage lapsed users# •  Think of this as placing filters / guards on a randomly chosen user population# Custom)Defined)User)Segments) Controlled)Experimenta=on)Panel) 8
  • 9. Key Components of an Experimentation Platform! Hashing function! Metrics – suite of KPI! Group)0) ! ! Revenue" ! ! F()))))))))))))) Group)1) ! ! Time"Spent" Abandonment" Click>Through"Rate" ! Group)NU1) ! ! ! Session"Length" Purchase"Rate" Logging! Dashboard! ! ! ! ! •  Detailed)logging)of)all)user)interac=ons) •  Metric)improvements)and)Sta=s=cal) Significance)in)a)central)place) Controlled)Experimenta=on)Panel) 9
  • 10. Ensure Identical Control and Treatment! Gender""•  Custom Segments# Male) Female)•  Frequency Distribution# CONTROL" TREAMENT" Region"Size" Small) Medium) Large) CONROL" TREATMENT" Prior"Exposure"•  Large Difference in Prior Exposure Rate violates δ%" assumptions# No) Yes) CONROL" TREATMENT" Controlled)Experimenta=on)Panel) 10
  • 11. A/A Tests!•  Run an experiment with two identical variants#•  Helps to determine if:# - Users are being split uniformly at random# - Correct data is being logged# - Variance between identical populations of users is acceptable#•  Challenge:! - Few purchases of high value deals render statistically significant difference between treatment and control# SPAIN"TRIP" $1,999" Controlled)Experimenta=on)Panel) 11
  • 12. Monitor Each Variant!•  Place yourself in each variant to validate the experience#!•  Wrong sort order!!!! Carefully)inspect)each)variant) Controlled)Experimenta=on)Panel) 12
  • 13. Objective Function!## Conversion" Revenue" P(conversion)) E(rev))=)P(conversion))*)price) ) ) •  Favors)lower) •  More)expensive)deals)can) price)deals) dominate) Need)to)balance)mul=ple,)oZen)conflic=ng)objec=ves)) Controlled)Experimenta=on)Panel) 13
  • 14. Measure Overall Impact!•  Test focuses on# - A particular area of the website# - A sub-population of users#•  Measure! - Improvement on the sub-segment AND# - Entire population!# Measure)overall)impact)to)guard)against)cannibaliza=on) Controlled)Experimenta=on)Panel) 14
  • 15. Panel Discussion: Questions! Controlled)Experimenta=on)Panel) 15
  • 16. Acknowledgements#Thanks to many talented individuals at Groupon I am privileged to work with!#•  Data Science#•  Engineering#•  Marketing / Market Research# Controlled)Experimenta=on)Panel) 16
  • 17. Rajesh Parekh! Groupon! rajesh@groupon.com! !Controlled)Experimenta=on)Panel) 17