Your SlideShare is downloading. ×
0
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Strata New York 2012: Continuous Experimentation with Continuous Deployment
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Strata New York 2012: Continuous Experimentation with Continuous Deployment

606

Published on

Continuous deployment is core to Etsy, and we push public facing code over 30 times per day. Evaluating an experiment amidst this shifting landscape is a difficult task as our traditional methods of …

Continuous deployment is core to Etsy, and we push public facing code over 30 times per day. Evaluating an experiment amidst this shifting landscape is a difficult task as our traditional methods of monitoring operational metrics don’t provide enough information to make product-level decisions. To this end, we have developed internal tooling for deep analytics that enables us to systematically analyze our experimental results in a continuously changing environment.

This talk will focus on the analysis framework that we have built – from the raw logging data, to our elastic mapreduce-based data transformations, to the final dashboards and underlying statistics that drive decision making.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
606
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
12
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  1. Continuous Experimentation with Continuous Deployment Steve Mardenfeld EtsyMonday, November 5, 2012
  2. Monday, November 5, 2012
  3. World’s Handmade MarketMonday, November 5, 2012
  4. Continuous DeploymentMonday, November 5, 2012
  5. Monday, November 5, 2012
  6. Monday, November 5, 2012
  7. MetricsMonday, November 5, 2012
  8. Graph FactoryMonday, November 5, 2012
  9. OopsMonday, November 5, 2012
  10. OopsMonday, November 5, 2012
  11. Good Enough?Monday, November 5, 2012
  12. What   is   Better?Monday, November 5, 2012
  13. Sometimes it’s ObviousMonday, November 5, 2012
  14. Most Times it’s NotMonday, November 5, 2012
  15. Most Times it’s Not Cart Page RestyledMonday, November 5, 2012
  16. Most Times it’s Not Etsy Mentioned on Oprah Cart Page Restyled Justin Beiber Internet Goes Sneezes Down in Brooklyn New Cat MemeMonday, November 5, 2012
  17. A/B Testing?Monday, November 5, 2012
  18. Monday, November 5, 2012
  19. Monday, November 5, 2012
  20. Test   Them   BothMonday, November 5, 2012
  21. Statistics?Monday, November 5, 2012
  22. First Pass HADOOPMonday, November 5, 2012
  23. First Pass CASCADINGMonday, November 5, 2012
  24. First Pass RMonday, November 5, 2012
  25. First Pass CASCADING HADOOP RMonday, November 5, 2012
  26. Automate It A/B AnalyzerMonday, November 5, 2012
  27. A/B Analyzer •Framework for looking at experiments •Done for every experiment beforehandMonday, November 5, 2012
  28. A/B AnalyzerMonday, November 5, 2012
  29. A/B AnalyzerMonday, November 5, 2012
  30. Monday, November 5, 2012
  31. 3 Types of Metrics •Co-occurence •visits that did X (searched) •average number of X (searches) •Funnels •Special Cases*Monday, November 5, 2012
  32. Co-Occurence •Co-occurence of all actions with the variant •i.e. - how many people saw this variant and then purchased •i.e. - average number of purchases per variantMonday, November 5, 2012
  33. Funnels Search Listing Buy BuyMonday, November 5, 2012
  34. Funnels Search Listing Buy Buy Home Search Listing BuyBuyMonday, November 5, 2012
  35. Funnels Search Listing Buy Buy Home Search Listing Buy BuyBuyMonday, November 5, 2012
  36. Special Cases •Exiting pages •Bounce rate •Visit lengthMonday, November 5, 2012
  37. Pipeline Logs A/B COLLECTION Client HDFS Events Visits MySql PROCESSING DBMonday, November 5, 2012
  38. Bucketing Home Search Listing Buy Search A/B TestMonday, November 5, 2012
  39. Bucketing Home Search Listing Buy Search A/B TestMonday, November 5, 2012
  40. Process •Idea •Hypothesis •Deploy •Wait •AnalysisMonday, November 5, 2012
  41. Process •Idea •Hypothesis •Deploy •Wait •AnalysisMonday, November 5, 2012
  42. IdeaMonday, November 5, 2012
  43. Changing SearchMonday, November 5, 2012
  44. Related Searches?Monday, November 5, 2012
  45. Process •Idea •Hypothesis •Deploy •Wait •AnalysisMonday, November 5, 2012
  46. Be Specific •What does an improvement look like? •More clicks? •Less searches? •Shorter visits? •Longer visits?Monday, November 5, 2012
  47. Bet Driven DevelopmentMonday, November 5, 2012
  48. Is this even worth it?Monday, November 5, 2012
  49. Process •Idea •Hypothesis •Deploy •Wait •AnalysisMonday, November 5, 2012
  50. Code  It  UpMonday, November 5, 2012
  51. CodeMonday, November 5, 2012
  52. Turn it OnMonday, November 5, 2012
  53. Process •Idea •Hypothesis •Deploy •Wait •AnalysisMonday, November 5, 2012
  54. Realtime?Monday, November 5, 2012
  55. WaitMonday, November 5, 2012
  56. Process •Idea •Hypothesis •Code •Wait •AnalysisMonday, November 5, 2012
  57. Making Sense of ResultsMonday, November 5, 2012
  58. Back To Related SearchesMonday, November 5, 2012
  59. Back To Related SearchesMonday, November 5, 2012
  60. Monday, November 5, 2012
  61. Monday, November 5, 2012
  62. Final Thoughts •Bet driven development - be explicit about your changes •Easy experimental setup and quick analysis makes for more experiments •An experiment is successful if you learned something, not if you launchedMonday, November 5, 2012
  63. Monday, November 5, 2012
  64. http://www.slideshare.net/powerlexis/retro-slides-from-ibmMonday, November 5, 2012

×