Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A/B Testing in the Wild


Published on

September 2017 presentation on A/B Testing at Metis' Demystifying Data Science conference.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

A/B Testing in the Wild

  1. 1. Emily Robinson Data Analyst, Etsy A/B Testing in the Wild
  2. 2. DISCLAIMER: This talk represents my own views, not those of Etsy
  3. 3. Etsy
  4. 4. Etsy is a global creative commerce platform. We build markets, services and economic opportunities for creative entrepreneurs. Etsy
  5. 5. Our Items
  6. 6. 1.8M active sellers AS OF JUNE 30, 2017 30.6M active buyers AS OF JUNE 30, 2017 $2.84B annual GMS IN 2016 45+M items for sale AS OF MARCH 31, 2017 Photo by Kirsty-Lyn Jameson By The Numbers
  7. 7. A/B Testing
  8. 8. What is A/B Testing?
  9. 9. Old Experience
  10. 10. New Experience
  11. 11. A/B Testing: It’s Everywhere
  12. 12. Highly Researched
  13. 13. My Perspective Millions of daily visitors Data Engineering Pipeline
  14. 14. Generating numbers is easy, generating numbers you should trust is hard!
  15. 15. • “Election surveys are done with a few thousand people”1 • Targeting small effects • A .5% change in conversion rate (e.g. 6% to 6.03%) on a high traffic page is millions of dollars annually 1Online Experimentation at Microsoft Why Statistics Anyway?
  16. 16. Statistical Challenges
  17. 17. Visit: activity by browser over a defined time period (30 minutes) Browser: Cookie or device ID (for apps) User: Signed-in user ID 1 2 3 Level of Analysis
  18. 18. I really want my own lightsaber Browser vs Visit: An Example
  19. 19. Next Day
  20. 20. BROWSERVISIT Independence violation assumption Cannibalization potential Tighter attribution Introduces noise Captures relevant later behavior Misses multiple events for proportion metrics Pros and Cons
  21. 21. Business Challenges
  22. 22. Working with Teams
  23. 23. Demonstrate Value: Prioritization, feasibility, sequencing 2 Early Involvement: No post-mortems 3 Proactive Communication Develop Relationship: Understand Teammates 1
  24. 24. Question: What is the conversion rate in Estonia on Saturday of users looking in the wedding category? First Response: What decision are you using this for? Dealing with Adhoc Questions
  25. 25. Helps Avoid This
  26. 26. Checks Translation
  27. 27. From Julia Evans, @b0rk “How to be a Wizard Programmer” And at the End of the Day …
  28. 28. • Jack Perkins, Anastasia Erbe, & Evan D’Agostini (former & fellow search analysts • Michael Berkowitz, Callie McRee, David Robinson, Bill Ulammandakh, & Dana-Levin Robinson (for presentation feedback) • Etsy Analytics team • Etsy Search Experience & Search Ranking teams Acknowledgments
  29. 29. EMILY ROBINSON | Data Analyst, Etsy Thank you! @Robinson_es