A/B Testing + Continuous Delivery = Everyday Product Launches

475
-1

Published on

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1v52uOK.

Nellwyn Thomas discusses how Etsy is using A/B testing, how Etsy's data-driven culture has evolved over time and how continuous delivery and big data can coexist. Filmed at qconnewyork.com.

Nellwyn Thomas is Director of Analytics at Etsy. She leads a team of data analysts who partner with product, marketing, and engineering to scout, build, instrument and improve Etsy's product portfolio. Before Etsy, she worked on analytics and product teams at two small NYC start-ups. She is a graduate of Harvard College, and holds a master's degree from the University of Pennsylvania.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
475
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

A/B Testing + Continuous Delivery = Everyday Product Launches

  1. 1. Continuous delivery, A/B testing + everyday launch decisions at Etsy QCon NY 2014 Nellwyn Thomas Director of Analytics, Etsy nellwyn@etsy.com @nellwyn
  2. 2. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /ab-testing-continuous-delivery InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month
  3. 3. Presented at QCon New York www.qconnewyork.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
  4. 4. etsy, briefly deploying experimenting making decisions, everyday takeaways !
  5. 5. etsy, briefly deploying experimenting making decisions, everyday takeaways
  6. 6. The world’s handmade and vintage marketplace.
  7. 7. Over 25M items from 1M shops in 150 countries.
  8. 8. $1.35B sales in 2013 !
  9. 9. 30 deploys by 200+ people per day ! ! ! ! This is continuous delivery at Etsy
  10. 10. 3 dozen experiments running simultaneously ! ! ! ! This is experimentation at Etsy
  11. 11. etsy, briefly deploying experimenting making decisions, everyday takeaways
  12. 12. Basic principles • A single source repository • Automate the build • Automate the testing • Automate deployment • Always ready to deploy quality software
  13. 13. Why continuous delivery? • Move faster • Fail faster • More empowering • Breeds confidence
  14. 14. Why continuous delivery? “Because you’re integrating so frequently, there is significantly less back-tracking to discover where things went wrong, so you can spend more time building features.” ! http://www.thoughtworks.com/continuous-integration
  15. 15. config.php template.tpl
  16. 16. CI Status (1 hour) End-User (1 hour) Error Logs (All) (1 hour) .................................................................. Deploy Dashboard Git Commits Princess Supergrep Supergrep Forum Watcher Time: 1 hour Until: Now Hide Deploys Deploys hidden: Config Web / Api / Atlas Search Photos Blog Chef (prod) Schemanator Push rbateman on qa 6m since 7, 2012 9:54 Three-Armed Sweaters error pages · historical clip Akamai Whoopsies origin failures · historical clip Edgecast Whoopsies origin failures · historical clip Screwed Users unique users · historical clip All File or Script Not Found current · historical clip Exceptions current · historical clip Fatals current · historical clip Errors current · historical clip Warnings current · historical clip Segmentation Faults current · historical clip PHP Allowed Memory Size Exhausted current · historical clip PHP Maximum Execution Time current · historical clip Memcache Connection Failures current · historical clip Thrift Exceptions current · historical clip
  17. 17. Segmentation Faults current · historical clip PHP Allowed Memory Other Errors (1 hour) Size Exhausted current · historical Business Graphs (1 hour) clip PHP Maximum Execution Time current · historical clip Memcache Connection Failures current · historical clip Thrift Exceptions current · historical clip StatsD packet receive errors · historical clip CSRF Mismatch current · historical clip CSRF Missing current · historical clip Client Side (JS) Errors current · historical clip Empty Image URLs grey.gif · Default Avatars clip Registrations facebook · facebook (historical) · other · other (historical) clip Logins (82.00%) logins · logins (historical) · errors · errors (historical) clip Checkouts (97.81%) success · failure clip Post Checkouts current · historical clip Mobile Checkouts iOS · mobile web clip Direct Checkout Checkouts (98.00%) success · failure clip
  18. 18. Post Checkouts current · historical clip Mobile Checkouts iOS · mobile web Page Performance by Site Section (1 hour) clip Direct Checkout Checkouts (98.00%) success · failure clip Listings Creation/Renewal New & Copied · Renewed clip Requests logged-out · logged-in clip CDN Requests Balance akamai · edgecast · direct-origin clip Search Ads Requests · Responses · Logged clip Forum Posts bugs forum · help forum clip Seller Registration (100.00%) success · failure clip PHP time (median) listing · shop · profile · search · homepage clip PHP time (95th percentile) listing · shop · profile · search · homepage clip webpagetest front-end time listing · shop · profile · search · homepage clip lognormal load time listing · shop · profile · search · homepage clip
  19. 19. Page Performance by Site Section (1 hour) PHP time (median) listing · shop · profile · search · homepage Statistics displayed represent signed in page requests. See Page Performance Dashboard for more detail. Web Cluster (1 hour) API Cluster (1 hour) clip PHP time (95th percentile) listing · shop · profile · search · homepage clip webpagetest front-end time listing · shop · profile · search · homepage clip lognormal load time listing · shop · profile · search · homepage clip
  20. 20. API Cluster (1 hour) Gearman (1 hour) Memcache (1 hour)
  21. 21. Memcache (1 hour) Master01 Database Activity (1 hour) DBShard Database Activity (1 hour)
  22. 22. DBShard Database Activity (1 hour) For more detailed shard information see the MySQL Shards Dashboard DBIndex Database Activity (1 hour)
  23. 23. DBIndex Database Activity (1 hour) Outgoing Bandwidth (1 hour)
  24. 24. etsy, briefly deploying experimenting making decisions, everyday takeaways
  25. 25. Experiments are easy • Experiments need large samples, internet “big data” is all about large samples • Visitors are free • Websites are built for iterative change, an experiment is an iterative change
  26. 26. And they help. • Fail faster • Fights confirmation bias • Forces you to tie product to goals (hypotheses!) • Helps you understand your users
  27. 27. Why experiment? “Because you’re integrating so frequently, there is significantly less back-tracking to discover where things went wrong, so you can spend more time building features.” ! http://www.thoughtworks.com/continuous-integration
  28. 28. How do experiment? exactly the same way we deploy. !
  29. 29. Bucketing listing page cart page checkout
  30. 30. Logging hadoop cluster mysql db internal dashboard
  31. 31. reusing config, automated bucketing, automated analysis & graphing ! with no additional effort !
  32. 32. on off
  33. 33. etsy, briefly deploying experimenting making decisions, everyday takeaways
  34. 34. Buy Sell Registry Community Blogs Mobile Gift Cards Help Reeggiisstteerr Siiggnn IInn Search for items and shops Seeaarrcchh Caarrtt Home › BayTowneLeatherUSA › IPAD Cases Leather Ipad Sleeve/ Case/ Envelope/ Folio for IPAD 2 or 3. Full Grain Veg Tan Leather. Handmade by BayTowneLeatherUSA. This luxurious/rugged and well built IPAD case will protect your IPAD in style. Or you can use it as a folio to carry your necessities . Luxurious enough for use at the office, and rugged enough for use on the trail. Made of beautiful - rugged 5/6 oz. Premium veg tan leather (which will break in beautifully) ; it will develop a wonderful natural patina over time as only full grain veg tan leather can do. (Gets better with age) ***I make these cases ONE BY ONE as orders are received*** Only 1 available Like this item? Add it to your favorites Favorite to revisit it later. $74.00 USD Add to Cart Favorite Shop BayTowneLeatherUSA BayTowneLeatherUSA Add shop to favorites See who favorites this shop 38 items Shop owner Roger Marion, Illinois, United States Favorites Circle Feedback: 27, 100% pos. Contact Item Add item to favorites See who favorites this item
  35. 35. All credit: John Allspaw
  36. 36. All credit: John Allspaw
  37. 37. All credit: John Allspaw
  38. 38. All credit: John Allspaw
  39. 39. All credit: John Allspaw
  40. 40. All credit: John Allspaw
  41. 41. All credit: John Allspaw
  42. 42. All credit: me change time direction
  43. 43. etsy, briefly deploying experimenting making decisions, everyday takeaways
  44. 44. validate early and often.
  45. 45. leverage existing workflows. “Changing behavior works only if it can be based on the existing ‘culture.’” Peter Drucker
  46. 46. don’t confuse conflate iteration with “tweaking pixels” the importance of the change with the size of the change
  47. 47. More Goodness John Allspaw on CD http://www.kitchensoap.com/category/ talks/ ! Mike Brittain on CD www.mikebrittain.com/blog/talks/ ! Dan McKinley on CE http://mcfunley.com/design-for-continuous- experimentation !
  48. 48. The End Learn more etsy.com/careers github.com/etsy ! Nellwyn Thomas nellwyn@etsy.com @nellwyn ! !
  49. 49. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/ab-testing- continuous-delivery

×