2. Emily Robinson
Data Analyst, Etsy
A/B Testing in the Wild

3. DISCLAIMER:
This talk represents my own
views, not those of Etsy

4. Etsy

5. Etsy is a global creative commerce platform. We build
markets, services and economic opportunities for creative
entrepreneurs.
Etsy

6. Our Items

7. 1.8M
active sellers
AS OF JUNE 30, 2017
30.6M
active buyers
AS OF JUNE 30, 2017
$2.84B
annual GMS
IN 2016
45+M
items for sale
AS OF MARCH 31, 2017
Photo by Kirsty-Lyn Jameson
By The Numbers

8. A/B Testing

9. What is A/B Testing?

10. Old Experience

11. New Experience

12. A/B Testing: It’s Everywhere

13. Highly Researched

14. My Perspective
Millions
of daily
visitors
Data
Engineering
Pipeline

15. Generating numbers is easy,
generating numbers you
should trust is hard!

16. • “Election surveys are done with a few thousand
people”1
• Targeting small effects
• A .5% change in conversion rate (e.g. 6% to 6.03%) on a
high traffic page is millions of dollars annually
1Online Experimentation at Microsoft
Why Statistics Anyway?

17. Statistical
Challenges

18. Visit:
activity by
browser over a
defined time
period (30
minutes)
Browser:
Cookie or device ID
(for apps)
User:
Signed-in
user ID
1 2 3
Level of Analysis

19. I really want my
own lightsaber
Browser vs Visit: An Example

21. Next Day

22. BROWSERVISIT
Independence
violation assumption
Cannibalization
potential
Tighter attribution
Introduces
noise
Captures relevant later
behavior
Misses multiple events for
proportion metrics
Pros and Cons

23. Business
Challenges

24. Working with Teams

25. Demonstrate
Value:
Prioritization,
feasibility,
sequencing
2
Early
Involvement:
No post-mortems
3
Proactive Communication
Develop
Relationship:
Understand
Teammates
1

26. Question:
What is the conversion rate in Estonia on Saturday of
users looking in the wedding category?
First Response:
What decision are you using this for?
Dealing with Adhoc Questions

27. Helps Avoid This

28. Checks Translation

29. From Julia Evans, @b0rk
“How to be a Wizard Programmer”
And at the End of the Day …

30. • Jack Perkins, Anastasia Erbe, & Evan D’Agostini (former
& fellow search analysts
• Michael Berkowitz, Callie McRee, David Robinson, Bill
Ulammandakh, & Dana-Levin Robinson (for presentation
feedback)
• Etsy Analytics team
• Etsy Search Experience & Search Ranking teams
Acknowledgments

31. EMILY ROBINSON | Data Analyst, Etsy
Thank you!
@Robinson_es
Robinsones.github.io