Fast and Good: Alternate Approaches to Quality at Etsy - STPCon fall 2011
Upcoming SlideShare
Loading in...5
×
 

Fast and Good: Alternate Approaches to Quality at Etsy - STPCon fall 2011

on

  • 3,059 views

 

Statistics

Views

Total Views
3,059
Views on SlideShare
3,026
Embed Views
33

Actions

Likes
2
Downloads
29
Comments
2

6 Embeds 33

http://coderwall.com 13
http://bitly.com 8
http://www.linkedin.com 6
https://twitter.com 4
http://a0.twimg.com 1
https://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

12 of 2

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • Here are a couple of brief video clips where I summarize my talk.
    http://bit.ly/uhEP1Q (thanks uTest!) and http://bit.ly/tgAfKa (thanks Search Software Quality!)
    Are you sure you want to
    Your message goes here
    Processing…
  • At Etsy we have spent the last several years learning to move fast at scale. Building on the precedents set by organizations like Flickr, IMVU and Facebook, we have created a testing methodology that is highly automated, low-ceremony and involves participation from individuals across the organization. As our development cycle became more rapid many challenges to testing arose: What is the exploratory tester’s role in such a fast-moving environment? How does one validate simultaneous releases by many different functional teams? How would the feedback loop between the tester and designer need to change? Many of the answers to these questions appeared counter-intuitive at first: never cutting release branches, eliminating the staging environment, asking engineers to assume responsibility for testing their own code. The end result is an online marketplace with hundreds of thousands of active sellers collectively drawing over a billion page views a month. This is the story of how the Etsy team learned to move fast, mitigate risk and deliver quality.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Fast and Good: Alternate Approaches to Quality at Etsy - STPCon fall 2011 Fast and Good: Alternate Approaches to Quality at Etsy - STPCon fall 2011 Presentation Transcript

    • fast and goodAlternative Approaches to Quality at Etsy Noah Sussman STPCon Fall 2011, Dallas TX #STPCon
    • Our mission is to empower people to change the waythe global economy works. We see a world in whichvery-very small businesses have much-much moresway in shaping the economy, local living economiesare thriving everywhere, and people value authorshipand provenance as much as price and convenience. Weare bringing heart to commerce and making the worldmore fair, more sustainable, and more fun.
    • September 2011: 489 Web deployments 110 people committed to trunk4,519 changesets in all 9.2 changesets average per production deployment
    • September 2011: 489 Web deployments 80 people deployed to production20 or fewer committers in each deployment5.7 committers average per deployment
    • 10 million registered members25 million unique visitors per month800,000 active sellers150 different countries10 million listed items$314 million in sales in 2010
    • In 2010, MTTR was 6.4 minutesfor severity one outages
    • Six change-related outages in 2010Out of thousands of production deployments
    • “All existing revision control systems were built bypeople who build installed software.” ~Paul Hammond
    • All canonical software life cycles were designed bypeople who build installed software
    • “In the online world… there is one consumer of thesoftware: you. The users are consumers of theservice built atop the software.” ~ Theo Schlossnagle
    • Optimize for recoveryrather than focusing onpreventing failure.Failure is inevitable.
    • Default to open access
    • No blame
    • Use your ownjudgement
    • “an airport without an air traffic controller” Chad Dickerson
    • faster deploys = smaller changesets = fewer change-related outages
    • Self-serve deployment
    • princessbleeding edge code,production data and services
    • Shared development environment Single dev database Everyone commits to trunk
    • low-ceremony process
    • A test is a design document first
    • Many small deployments, many small tests
    • Beyond developer testing1. Test planning2. GUI-driven regression testing3. Performance testing4. Security / Penetration testing5. Nagios
    • Make it easy to write and run tests
    • Make it painless to use fixtures and mocks
    • Most bugs, most of the time, are easily nailed given evenan incomplete but suggestive characterization of theirerror conditions at source-code level. ~ESR, “How Many Eyeballs Tame Complexity,” 1997
    • Debugging is twice as hardas writing the code in thefirst place. Therefore, if youwrite the code as cleverly aspossible, you are, bydefinition, not smart enoughto debug it. ~ Brian Kernighan
    • Test to address technical debt
    • Keep the feedback loop short
    • Graphmonitorit didn’t happenAt Etsy we or over 40,000 real-time metrics
    • Business Metrics
    • Application Metrics
    • System Metrics
    • We hire our users
    • Tests are one of the tools we use todetect and prevent failure
    • There will always be bugs in production
    • Unexpected failures tend to be catastrophic
    • Anomaly detection is hard
    • Just ship
    • How do you deploy large features?
    • Releasing a feature is decoupled from deploying code
    • if ($cfg["new_search"]) { // new hotness $resp = search_solr();} else { // old busted $resp = search_grep();}
    • $cfg = array( checkout => true, homepage => true, profiles => true, new_search => false,);
    • Dark launchesAdmin-only rolloutsA/B campaignsetsy.com/prototypesWe are always testing
    • A culture of responsibility
    • A community of artisans
    • You are a participant observer
    • etsy.com/careers
    • Noah Sussmannoah@etsy.com@noahsussman
    • photo creditspenguins: paulmcdee http://www.flickr.com/photos/paulmcdee/2806715569aerial photograph of snowy road: K. Leidorf http://leidorf.blogspot.com/search/label/20D03855death in the mirror: Richard Avedon for the New YorkerWomen at work on C-47 Douglas cargo transport http://www.flickr.com/photos/library_of_congress/2179137415/in/photostream/Woman aircraft worker http://www.flickr.com/photos/library_of_congress/2179930812/cool bike with wooden handlebars: Bertelli http://www.bertellibici.com/product.php?works_id=112clover keys: Brian Cerveny http://www.flickr.com/photos/briancerveny/48889960/in/photostream/Etsy office monitor wall and Etsy bike rack: Mashables Cubicle Spy http://mashable.com/2010/10/13/etsy-office-picshorse fly: Lukjonis http://www.flickr.com/photos/38628972@N05/5884111163/in/photostream/Gandalf vs balrog: the brothers HildebrandtJust Ship t-shirt: Kellan http://www.etsy.com/shop/kellanem/soldbarn raising: dogpose http://www.flickr.com/photos/8371074@N05/2865798475/model train coupling: David E. Smith http://www.flickr.com/photos/david_e_smith/3566697122/sizes/l/in/photostream/If I used your image and didnt credit you, its because I couldnt figure out who you were -- email me and I will gladly credit you.If you object to my using an image to which you own the rights, please contact me and I will remove that image from the deck.