Your SlideShare is downloading. ×
Continuous delivery, 
A/B testing + everyday 
launch decisions at Etsy 
QCon NY 2014 
Nellwyn Thomas 
Director of Analytic...
Watch the video with slide 
synchronization on InfoQ.com! 
http://www.infoq.com/presentations 
/ab-testing-continuous-deli...
Presented at QCon New York 
www.qconnewyork.com 
Purpose of QCon 
- to empower software development by facilitating the sp...
etsy, briefly 
deploying 
experimenting 
making decisions, everyday 
takeaways 
!
etsy, briefly 
deploying 
experimenting 
making decisions, everyday 
takeaways
The world’s handmade and vintage marketplace.
Over 25M items from 1M shops in 150 countries.
$1.35B sales in 2013 
!
30 deploys by 200+ people per day 
! 
! 
! 
! 
This is continuous delivery at Etsy
3 dozen experiments running simultaneously 
! 
! 
! 
! 
This is experimentation at Etsy
etsy, briefly 
deploying 
experimenting 
making decisions, everyday 
takeaways
Basic principles 
• A single source repository 
• Automate the build 
• Automate the testing 
• Automate deployment 
• Alw...
Why continuous delivery? 
• Move faster 
• Fail faster 
• More empowering 
• Breeds confidence
Why continuous delivery? 
“Because you’re integrating so frequently, 
there is significantly less back-tracking to 
discov...
config.php 
template.tpl
CI Status (1 hour) 
End-User (1 hour) 
Error Logs (All) (1 hour) 
...........................................................
Segmentation Faults 
current · historical 
clip PHP Allowed Memory 
Other Errors (1 hour) 
Size Exhausted 
current · histo...
Post Checkouts 
current · historical 
clip Mobile Checkouts 
iOS · mobile web 
Page Performance by Site Section (1 hour) 
...
Page Performance by Site Section (1 hour) 
PHP time (median) 
listing · shop · profile · search · homepage 
Statistics dis...
API Cluster (1 hour) 
Gearman (1 hour) 
Memcache (1 hour)
Memcache (1 hour) 
Master01 Database Activity (1 hour) 
DBShard Database Activity (1 hour)
DBShard Database Activity (1 hour) 
For more detailed shard information see the MySQL Shards Dashboard 
DBIndex Database A...
DBIndex Database Activity (1 hour) 
Outgoing Bandwidth (1 hour)
etsy, briefly 
deploying 
experimenting 
making decisions, everyday 
takeaways
Experiments are easy 
• Experiments need large samples, internet “big 
data” is all about large samples 
• Visitors are fr...
And they help. 
• Fail faster 
• Fights confirmation bias 
• Forces you to tie product to goals (hypotheses!) 
• Helps you...
Why experiment? 
“Because you’re integrating so frequently, 
there is significantly less back-tracking to 
discover where ...
How do experiment? 
exactly the same way we deploy. 
!
Bucketing 
listing 
page 
cart 
page checkout
Logging 
hadoop cluster 
mysql db 
internal 
dashboard
reusing config, 
automated bucketing, 
automated analysis & graphing 
! 
with no additional effort 
!
on 
off
etsy, briefly 
deploying 
experimenting 
making decisions, everyday 
takeaways
Buy Sell Registry Community Blogs Mobile Gift Cards Help 
Reeggiisstteerr Siiggnn IInn Search for items and shops Seeaarrc...
All credit: John Allspaw
All credit: John Allspaw
All credit: John Allspaw
All credit: John Allspaw
All credit: John Allspaw
All credit: John Allspaw
All credit: John Allspaw
All credit: me 
change 
time 
direction
etsy, briefly 
deploying 
experimenting 
making decisions, everyday 
takeaways
validate early 
and often.
leverage 
existing 
workflows. 
“Changing behavior works only if it 
can be based on the existing ‘culture.’” 
Peter Druck...
don’t confuse 
conflate 
iteration with 
“tweaking pixels” 
the importance 
of the change 
with the size of 
the change
More Goodness 
John Allspaw on CD 
http://www.kitchensoap.com/category/ 
talks/ 
! 
Mike Brittain on CD 
www.mikebrittain....
The End 
Learn more 
etsy.com/careers 
github.com/etsy 
! 
Nellwyn Thomas 
nellwyn@etsy.com 
@nellwyn 
! 
!
Watch the video with slide synchronization on 
InfoQ.com! 
http://www.infoq.com/presentations/ab-testing- 
continuous-deli...
A/B Testing + Continuous Delivery = Everyday Product Launches
A/B Testing + Continuous Delivery = Everyday Product Launches
A/B Testing + Continuous Delivery = Everyday Product Launches
A/B Testing + Continuous Delivery = Everyday Product Launches
A/B Testing + Continuous Delivery = Everyday Product Launches
A/B Testing + Continuous Delivery = Everyday Product Launches
A/B Testing + Continuous Delivery = Everyday Product Launches
A/B Testing + Continuous Delivery = Everyday Product Launches
A/B Testing + Continuous Delivery = Everyday Product Launches
A/B Testing + Continuous Delivery = Everyday Product Launches
A/B Testing + Continuous Delivery = Everyday Product Launches
A/B Testing + Continuous Delivery = Everyday Product Launches
A/B Testing + Continuous Delivery = Everyday Product Launches
A/B Testing + Continuous Delivery = Everyday Product Launches
A/B Testing + Continuous Delivery = Everyday Product Launches
A/B Testing + Continuous Delivery = Everyday Product Launches
A/B Testing + Continuous Delivery = Everyday Product Launches
A/B Testing + Continuous Delivery = Everyday Product Launches
A/B Testing + Continuous Delivery = Everyday Product Launches
A/B Testing + Continuous Delivery = Everyday Product Launches
A/B Testing + Continuous Delivery = Everyday Product Launches
A/B Testing + Continuous Delivery = Everyday Product Launches
A/B Testing + Continuous Delivery = Everyday Product Launches
A/B Testing + Continuous Delivery = Everyday Product Launches
A/B Testing + Continuous Delivery = Everyday Product Launches
A/B Testing + Continuous Delivery = Everyday Product Launches
Upcoming SlideShare
Loading in...5
×

A/B Testing + Continuous Delivery = Everyday Product Launches

312

Published on

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1v52uOK.

Nellwyn Thomas discusses how Etsy is using A/B testing, how Etsy's data-driven culture has evolved over time and how continuous delivery and big data can coexist. Filmed at qconnewyork.com.

Nellwyn Thomas is Director of Analytics at Etsy. She leads a team of data analysts who partner with product, marketing, and engineering to scout, build, instrument and improve Etsy's product portfolio. Before Etsy, she worked on analytics and product teams at two small NYC start-ups. She is a graduate of Harvard College, and holds a master's degree from the University of Pennsylvania.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
312
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "A/B Testing + Continuous Delivery = Everyday Product Launches"

  1. 1. Continuous delivery, A/B testing + everyday launch decisions at Etsy QCon NY 2014 Nellwyn Thomas Director of Analytics, Etsy nellwyn@etsy.com @nellwyn
  2. 2. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /ab-testing-continuous-delivery InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month
  3. 3. Presented at QCon New York www.qconnewyork.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
  4. 4. etsy, briefly deploying experimenting making decisions, everyday takeaways !
  5. 5. etsy, briefly deploying experimenting making decisions, everyday takeaways
  6. 6. The world’s handmade and vintage marketplace.
  7. 7. Over 25M items from 1M shops in 150 countries.
  8. 8. $1.35B sales in 2013 !
  9. 9. 30 deploys by 200+ people per day ! ! ! ! This is continuous delivery at Etsy
  10. 10. 3 dozen experiments running simultaneously ! ! ! ! This is experimentation at Etsy
  11. 11. etsy, briefly deploying experimenting making decisions, everyday takeaways
  12. 12. Basic principles • A single source repository • Automate the build • Automate the testing • Automate deployment • Always ready to deploy quality software
  13. 13. Why continuous delivery? • Move faster • Fail faster • More empowering • Breeds confidence
  14. 14. Why continuous delivery? “Because you’re integrating so frequently, there is significantly less back-tracking to discover where things went wrong, so you can spend more time building features.” ! http://www.thoughtworks.com/continuous-integration
  15. 15. config.php template.tpl
  16. 16. CI Status (1 hour) End-User (1 hour) Error Logs (All) (1 hour) .................................................................. Deploy Dashboard Git Commits Princess Supergrep Supergrep Forum Watcher Time: 1 hour Until: Now Hide Deploys Deploys hidden: Config Web / Api / Atlas Search Photos Blog Chef (prod) Schemanator Push rbateman on qa 6m since 7, 2012 9:54 Three-Armed Sweaters error pages · historical clip Akamai Whoopsies origin failures · historical clip Edgecast Whoopsies origin failures · historical clip Screwed Users unique users · historical clip All File or Script Not Found current · historical clip Exceptions current · historical clip Fatals current · historical clip Errors current · historical clip Warnings current · historical clip Segmentation Faults current · historical clip PHP Allowed Memory Size Exhausted current · historical clip PHP Maximum Execution Time current · historical clip Memcache Connection Failures current · historical clip Thrift Exceptions current · historical clip
  17. 17. Segmentation Faults current · historical clip PHP Allowed Memory Other Errors (1 hour) Size Exhausted current · historical Business Graphs (1 hour) clip PHP Maximum Execution Time current · historical clip Memcache Connection Failures current · historical clip Thrift Exceptions current · historical clip StatsD packet receive errors · historical clip CSRF Mismatch current · historical clip CSRF Missing current · historical clip Client Side (JS) Errors current · historical clip Empty Image URLs grey.gif · Default Avatars clip Registrations facebook · facebook (historical) · other · other (historical) clip Logins (82.00%) logins · logins (historical) · errors · errors (historical) clip Checkouts (97.81%) success · failure clip Post Checkouts current · historical clip Mobile Checkouts iOS · mobile web clip Direct Checkout Checkouts (98.00%) success · failure clip
  18. 18. Post Checkouts current · historical clip Mobile Checkouts iOS · mobile web Page Performance by Site Section (1 hour) clip Direct Checkout Checkouts (98.00%) success · failure clip Listings Creation/Renewal New & Copied · Renewed clip Requests logged-out · logged-in clip CDN Requests Balance akamai · edgecast · direct-origin clip Search Ads Requests · Responses · Logged clip Forum Posts bugs forum · help forum clip Seller Registration (100.00%) success · failure clip PHP time (median) listing · shop · profile · search · homepage clip PHP time (95th percentile) listing · shop · profile · search · homepage clip webpagetest front-end time listing · shop · profile · search · homepage clip lognormal load time listing · shop · profile · search · homepage clip
  19. 19. Page Performance by Site Section (1 hour) PHP time (median) listing · shop · profile · search · homepage Statistics displayed represent signed in page requests. See Page Performance Dashboard for more detail. Web Cluster (1 hour) API Cluster (1 hour) clip PHP time (95th percentile) listing · shop · profile · search · homepage clip webpagetest front-end time listing · shop · profile · search · homepage clip lognormal load time listing · shop · profile · search · homepage clip
  20. 20. API Cluster (1 hour) Gearman (1 hour) Memcache (1 hour)
  21. 21. Memcache (1 hour) Master01 Database Activity (1 hour) DBShard Database Activity (1 hour)
  22. 22. DBShard Database Activity (1 hour) For more detailed shard information see the MySQL Shards Dashboard DBIndex Database Activity (1 hour)
  23. 23. DBIndex Database Activity (1 hour) Outgoing Bandwidth (1 hour)
  24. 24. etsy, briefly deploying experimenting making decisions, everyday takeaways
  25. 25. Experiments are easy • Experiments need large samples, internet “big data” is all about large samples • Visitors are free • Websites are built for iterative change, an experiment is an iterative change
  26. 26. And they help. • Fail faster • Fights confirmation bias • Forces you to tie product to goals (hypotheses!) • Helps you understand your users
  27. 27. Why experiment? “Because you’re integrating so frequently, there is significantly less back-tracking to discover where things went wrong, so you can spend more time building features.” ! http://www.thoughtworks.com/continuous-integration
  28. 28. How do experiment? exactly the same way we deploy. !
  29. 29. Bucketing listing page cart page checkout
  30. 30. Logging hadoop cluster mysql db internal dashboard
  31. 31. reusing config, automated bucketing, automated analysis & graphing ! with no additional effort !
  32. 32. on off
  33. 33. etsy, briefly deploying experimenting making decisions, everyday takeaways
  34. 34. Buy Sell Registry Community Blogs Mobile Gift Cards Help Reeggiisstteerr Siiggnn IInn Search for items and shops Seeaarrcchh Caarrtt Home › BayTowneLeatherUSA › IPAD Cases Leather Ipad Sleeve/ Case/ Envelope/ Folio for IPAD 2 or 3. Full Grain Veg Tan Leather. Handmade by BayTowneLeatherUSA. This luxurious/rugged and well built IPAD case will protect your IPAD in style. Or you can use it as a folio to carry your necessities . Luxurious enough for use at the office, and rugged enough for use on the trail. Made of beautiful - rugged 5/6 oz. Premium veg tan leather (which will break in beautifully) ; it will develop a wonderful natural patina over time as only full grain veg tan leather can do. (Gets better with age) ***I make these cases ONE BY ONE as orders are received*** Only 1 available Like this item? Add it to your favorites Favorite to revisit it later. $74.00 USD Add to Cart Favorite Shop BayTowneLeatherUSA BayTowneLeatherUSA Add shop to favorites See who favorites this shop 38 items Shop owner Roger Marion, Illinois, United States Favorites Circle Feedback: 27, 100% pos. Contact Item Add item to favorites See who favorites this item
  35. 35. All credit: John Allspaw
  36. 36. All credit: John Allspaw
  37. 37. All credit: John Allspaw
  38. 38. All credit: John Allspaw
  39. 39. All credit: John Allspaw
  40. 40. All credit: John Allspaw
  41. 41. All credit: John Allspaw
  42. 42. All credit: me change time direction
  43. 43. etsy, briefly deploying experimenting making decisions, everyday takeaways
  44. 44. validate early and often.
  45. 45. leverage existing workflows. “Changing behavior works only if it can be based on the existing ‘culture.’” Peter Drucker
  46. 46. don’t confuse conflate iteration with “tweaking pixels” the importance of the change with the size of the change
  47. 47. More Goodness John Allspaw on CD http://www.kitchensoap.com/category/ talks/ ! Mike Brittain on CD www.mikebrittain.com/blog/talks/ ! Dan McKinley on CE http://mcfunley.com/design-for-continuous- experimentation !
  48. 48. The End Learn more etsy.com/careers github.com/etsy ! Nellwyn Thomas nellwyn@etsy.com @nellwyn ! !
  49. 49. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/ab-testing- continuous-delivery

×