Continuous                           Experimentation                           with Continuous                            ...
Monday, November 5, 2012
World’s Handmade MarketMonday, November 5, 2012
Continuous DeploymentMonday, November 5, 2012
Monday, November 5, 2012
Monday, November 5, 2012
MetricsMonday, November 5, 2012
Graph FactoryMonday, November 5, 2012
OopsMonday, November 5, 2012
OopsMonday, November 5, 2012
Good Enough?Monday, November 5, 2012
What	                             is	                             Better?Monday, November 5, 2012
Sometimes it’s ObviousMonday, November 5, 2012
Most Times it’s NotMonday, November 5, 2012
Most Times it’s Not                                                 Cart Page                                             ...
Most Times it’s Not     Etsy Mentioned        on Oprah                                 Cart Page                          ...
A/B Testing?Monday, November 5, 2012
Monday, November 5, 2012
Monday, November 5, 2012
Test	                             Them	                             BothMonday, November 5, 2012
Statistics?Monday, November 5, 2012
First Pass                           HADOOPMonday, November 5, 2012
First Pass                             CASCADINGMonday, November 5, 2012
First Pass                                    RMonday, November 5, 2012
First Pass                                    CASCADING                           HADOOP               RMonday, November 5...
Automate It                             A/B AnalyzerMonday, November 5, 2012
A/B Analyzer                       •Framework for looking at experiments                       •Done for every experiment ...
A/B AnalyzerMonday, November 5, 2012
A/B AnalyzerMonday, November 5, 2012
Monday, November 5, 2012
3 Types of Metrics                       •Co-occurence                        •visits that did X (searched)               ...
Co-Occurence                       •Co-occurence of all actions with the variant                        •i.e. - how many p...
Funnels                           Search     Listing   Buy                                                  BuyMonday, Nov...
Funnels                                  Search            Listing             Buy                                        ...
Funnels                                  Search            Listing             Buy                                        ...
Special Cases                       •Exiting pages                       •Bounce rate                       •Visit lengthM...
Pipeline                            Logs                          A/B           COLLECTION                   Client   HDFS...
Bucketing                           Home       Search        Listing   Buy                                  Search A/B Tes...
Bucketing                           Home       Search        Listing   Buy                                  Search A/B Tes...
Process                       •Idea                       •Hypothesis                       •Deploy                       ...
Process                       •Idea                       •Hypothesis                       •Deploy                       ...
IdeaMonday, November 5, 2012
Changing SearchMonday, November 5, 2012
Related Searches?Monday, November 5, 2012
Process                       •Idea                       •Hypothesis                       •Deploy                       ...
Be Specific                       •What does an improvement look like?                        •More clicks?               ...
Bet Driven DevelopmentMonday, November 5, 2012
Is this even worth it?Monday, November 5, 2012
Process                       •Idea                       •Hypothesis                       •Deploy                       ...
Code	  It	  UpMonday, November 5, 2012
CodeMonday, November 5, 2012
Turn it OnMonday, November 5, 2012
Process                       •Idea                       •Hypothesis                       •Deploy                       ...
Realtime?Monday, November 5, 2012
WaitMonday, November 5, 2012
Process                       •Idea                       •Hypothesis                       •Code                       •W...
Making Sense of ResultsMonday, November 5, 2012
Back To Related SearchesMonday, November 5, 2012
Back To Related SearchesMonday, November 5, 2012
Monday, November 5, 2012
Monday, November 5, 2012
Final Thoughts                       •Bet driven development - be explicit about your                           changes   ...
Monday, November 5, 2012
http://www.slideshare.net/powerlexis/retro-slides-from-ibmMonday, November 5, 2012
Upcoming SlideShare
Loading in …5
×

Strata New York 2012: Continuous Experimentation with Continuous Deployment

852 views
776 views

Published on

Continuous deployment is core to Etsy, and we push public facing code over 30 times per day. Evaluating an experiment amidst this shifting landscape is a difficult task as our traditional methods of monitoring operational metrics don’t provide enough information to make product-level decisions. To this end, we have developed internal tooling for deep analytics that enables us to systematically analyze our experimental results in a continuously changing environment.

This talk will focus on the analysis framework that we have built – from the raw logging data, to our elastic mapreduce-based data transformations, to the final dashboards and underlying statistics that drive decision making.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
852
On SlideShare
0
From Embeds
0
Number of Embeds
34
Actions
Shares
0
Downloads
12
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Strata New York 2012: Continuous Experimentation with Continuous Deployment

  1. 1. Continuous Experimentation with Continuous Deployment Steve Mardenfeld EtsyMonday, November 5, 2012
  2. 2. Monday, November 5, 2012
  3. 3. World’s Handmade MarketMonday, November 5, 2012
  4. 4. Continuous DeploymentMonday, November 5, 2012
  5. 5. Monday, November 5, 2012
  6. 6. Monday, November 5, 2012
  7. 7. MetricsMonday, November 5, 2012
  8. 8. Graph FactoryMonday, November 5, 2012
  9. 9. OopsMonday, November 5, 2012
  10. 10. OopsMonday, November 5, 2012
  11. 11. Good Enough?Monday, November 5, 2012
  12. 12. What   is   Better?Monday, November 5, 2012
  13. 13. Sometimes it’s ObviousMonday, November 5, 2012
  14. 14. Most Times it’s NotMonday, November 5, 2012
  15. 15. Most Times it’s Not Cart Page RestyledMonday, November 5, 2012
  16. 16. Most Times it’s Not Etsy Mentioned on Oprah Cart Page Restyled Justin Beiber Internet Goes Sneezes Down in Brooklyn New Cat MemeMonday, November 5, 2012
  17. 17. A/B Testing?Monday, November 5, 2012
  18. 18. Monday, November 5, 2012
  19. 19. Monday, November 5, 2012
  20. 20. Test   Them   BothMonday, November 5, 2012
  21. 21. Statistics?Monday, November 5, 2012
  22. 22. First Pass HADOOPMonday, November 5, 2012
  23. 23. First Pass CASCADINGMonday, November 5, 2012
  24. 24. First Pass RMonday, November 5, 2012
  25. 25. First Pass CASCADING HADOOP RMonday, November 5, 2012
  26. 26. Automate It A/B AnalyzerMonday, November 5, 2012
  27. 27. A/B Analyzer •Framework for looking at experiments •Done for every experiment beforehandMonday, November 5, 2012
  28. 28. A/B AnalyzerMonday, November 5, 2012
  29. 29. A/B AnalyzerMonday, November 5, 2012
  30. 30. Monday, November 5, 2012
  31. 31. 3 Types of Metrics •Co-occurence •visits that did X (searched) •average number of X (searches) •Funnels •Special Cases*Monday, November 5, 2012
  32. 32. Co-Occurence •Co-occurence of all actions with the variant •i.e. - how many people saw this variant and then purchased •i.e. - average number of purchases per variantMonday, November 5, 2012
  33. 33. Funnels Search Listing Buy BuyMonday, November 5, 2012
  34. 34. Funnels Search Listing Buy Buy Home Search Listing BuyBuyMonday, November 5, 2012
  35. 35. Funnels Search Listing Buy Buy Home Search Listing Buy BuyBuyMonday, November 5, 2012
  36. 36. Special Cases •Exiting pages •Bounce rate •Visit lengthMonday, November 5, 2012
  37. 37. Pipeline Logs A/B COLLECTION Client HDFS Events Visits MySql PROCESSING DBMonday, November 5, 2012
  38. 38. Bucketing Home Search Listing Buy Search A/B TestMonday, November 5, 2012
  39. 39. Bucketing Home Search Listing Buy Search A/B TestMonday, November 5, 2012
  40. 40. Process •Idea •Hypothesis •Deploy •Wait •AnalysisMonday, November 5, 2012
  41. 41. Process •Idea •Hypothesis •Deploy •Wait •AnalysisMonday, November 5, 2012
  42. 42. IdeaMonday, November 5, 2012
  43. 43. Changing SearchMonday, November 5, 2012
  44. 44. Related Searches?Monday, November 5, 2012
  45. 45. Process •Idea •Hypothesis •Deploy •Wait •AnalysisMonday, November 5, 2012
  46. 46. Be Specific •What does an improvement look like? •More clicks? •Less searches? •Shorter visits? •Longer visits?Monday, November 5, 2012
  47. 47. Bet Driven DevelopmentMonday, November 5, 2012
  48. 48. Is this even worth it?Monday, November 5, 2012
  49. 49. Process •Idea •Hypothesis •Deploy •Wait •AnalysisMonday, November 5, 2012
  50. 50. Code  It  UpMonday, November 5, 2012
  51. 51. CodeMonday, November 5, 2012
  52. 52. Turn it OnMonday, November 5, 2012
  53. 53. Process •Idea •Hypothesis •Deploy •Wait •AnalysisMonday, November 5, 2012
  54. 54. Realtime?Monday, November 5, 2012
  55. 55. WaitMonday, November 5, 2012
  56. 56. Process •Idea •Hypothesis •Code •Wait •AnalysisMonday, November 5, 2012
  57. 57. Making Sense of ResultsMonday, November 5, 2012
  58. 58. Back To Related SearchesMonday, November 5, 2012
  59. 59. Back To Related SearchesMonday, November 5, 2012
  60. 60. Monday, November 5, 2012
  61. 61. Monday, November 5, 2012
  62. 62. Final Thoughts •Bet driven development - be explicit about your changes •Easy experimental setup and quick analysis makes for more experiments •An experiment is successful if you learned something, not if you launchedMonday, November 5, 2012
  63. 63. Monday, November 5, 2012
  64. 64. http://www.slideshare.net/powerlexis/retro-slides-from-ibmMonday, November 5, 2012

×