Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Emily Robinson
Data Analyst, Etsy
A/B Testing in the Wild
DISCLAIMER:
This talk represents my own
views, not those of Etsy
Etsy
Etsy is a global creative commerce platform. We build
markets, services and economic opportunities for creative
entreprene...
Our Items
1.8M
active sellers
AS OF JUNE 30, 2017
30.6M
active buyers
AS OF JUNE 30, 2017
$2.84B
annual GMS
IN 2016
45+M
items for s...
A/B Testing
What is A/B Testing?
Old Experience
New Experience
A/B Testing: It’s Everywhere
Highly Researched
My Perspective
Millions
of daily
visitors
Data
Engineering
Pipeline
Generating numbers is easy,
generating numbers you
should trust is hard!
• “Election surveys are done with a few thousand
people”1
• Targeting small effects
• A .5% change in conversion rate (e.g...
Statistical
Challenges
Visit:
activity by
browser over a
defined time
period (30
minutes)
Browser:
Cookie or device ID
(for apps)
User:
Signed-in...
I really want my
own lightsaber
Browser vs Visit: An Example
Next Day
BROWSERVISIT
Independence
violation assumption
Cannibalization
potential
Tighter attribution
Introduces
noise
Captures rel...
Business
Challenges
Working with Teams
Demonstrate
Value:
Prioritization,
feasibility,
sequencing
2
Early
Involvement:
No post-mortems
3
Proactive Communication
...
Question:
What is the conversion rate in Estonia on Saturday of
users looking in the wedding category?
First Response:
Wha...
Helps Avoid This
Checks Translation
From Julia Evans, @b0rk
“How to be a Wizard Programmer”
And at the End of the Day …
• Jack Perkins, Anastasia Erbe, & Evan D’Agostini (former
& fellow search analysts
• Michael Berkowitz, Callie McRee, Davi...
EMILY ROBINSON | Data Analyst, Etsy
Thank you!
@Robinson_es
Robinsones.github.io
A/B Testing in the Wild
A/B Testing in the Wild
Upcoming SlideShare
Loading in …5
×

A/B Testing in the Wild

37 views

Published on

September 2017 presentation on A/B Testing at Metis' Demystifying Data Science conference.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

A/B Testing in the Wild

  1. 1. Emily Robinson Data Analyst, Etsy A/B Testing in the Wild
  2. 2. DISCLAIMER: This talk represents my own views, not those of Etsy
  3. 3. Etsy
  4. 4. Etsy is a global creative commerce platform. We build markets, services and economic opportunities for creative entrepreneurs. Etsy
  5. 5. Our Items
  6. 6. 1.8M active sellers AS OF JUNE 30, 2017 30.6M active buyers AS OF JUNE 30, 2017 $2.84B annual GMS IN 2016 45+M items for sale AS OF MARCH 31, 2017 Photo by Kirsty-Lyn Jameson By The Numbers
  7. 7. A/B Testing
  8. 8. What is A/B Testing?
  9. 9. Old Experience
  10. 10. New Experience
  11. 11. A/B Testing: It’s Everywhere
  12. 12. Highly Researched
  13. 13. My Perspective Millions of daily visitors Data Engineering Pipeline
  14. 14. Generating numbers is easy, generating numbers you should trust is hard!
  15. 15. • “Election surveys are done with a few thousand people”1 • Targeting small effects • A .5% change in conversion rate (e.g. 6% to 6.03%) on a high traffic page is millions of dollars annually 1Online Experimentation at Microsoft Why Statistics Anyway?
  16. 16. Statistical Challenges
  17. 17. Visit: activity by browser over a defined time period (30 minutes) Browser: Cookie or device ID (for apps) User: Signed-in user ID 1 2 3 Level of Analysis
  18. 18. I really want my own lightsaber Browser vs Visit: An Example
  19. 19. Next Day
  20. 20. BROWSERVISIT Independence violation assumption Cannibalization potential Tighter attribution Introduces noise Captures relevant later behavior Misses multiple events for proportion metrics Pros and Cons
  21. 21. Business Challenges
  22. 22. Working with Teams
  23. 23. Demonstrate Value: Prioritization, feasibility, sequencing 2 Early Involvement: No post-mortems 3 Proactive Communication Develop Relationship: Understand Teammates 1
  24. 24. Question: What is the conversion rate in Estonia on Saturday of users looking in the wedding category? First Response: What decision are you using this for? Dealing with Adhoc Questions
  25. 25. Helps Avoid This
  26. 26. Checks Translation
  27. 27. From Julia Evans, @b0rk “How to be a Wizard Programmer” And at the End of the Day …
  28. 28. • Jack Perkins, Anastasia Erbe, & Evan D’Agostini (former & fellow search analysts • Michael Berkowitz, Callie McRee, David Robinson, Bill Ulammandakh, & Dana-Levin Robinson (for presentation feedback) • Etsy Analytics team • Etsy Search Experience & Search Ranking teams Acknowledgments
  29. 29. EMILY ROBINSON | Data Analyst, Etsy Thank you! @Robinson_es Robinsones.github.io

×