Fast and Good: Alternate Approaches to Quality at Etsy - STPCon fall 2011
Upcoming SlideShare
Loading in...5

Fast and Good: Alternate Approaches to Quality at Etsy - STPCon fall 2011






Total Views
Views on SlideShare
Embed Views



6 Embeds 33 13 8 6 4 1 1



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.


12 of 2

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • Here are a couple of brief video clips where I summarize my talk. (thanks uTest!) and (thanks Search Software Quality!)
    Are you sure you want to
    Your message goes here
  • At Etsy we have spent the last several years learning to move fast at scale. Building on the precedents set by organizations like Flickr, IMVU and Facebook, we have created a testing methodology that is highly automated, low-ceremony and involves participation from individuals across the organization. As our development cycle became more rapid many challenges to testing arose: What is the exploratory tester’s role in such a fast-moving environment? How does one validate simultaneous releases by many different functional teams? How would the feedback loop between the tester and designer need to change? Many of the answers to these questions appeared counter-intuitive at first: never cutting release branches, eliminating the staging environment, asking engineers to assume responsibility for testing their own code. The end result is an online marketplace with hundreds of thousands of active sellers collectively drawing over a billion page views a month. This is the story of how the Etsy team learned to move fast, mitigate risk and deliver quality.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    Fast and Good: Alternate Approaches to Quality at Etsy - STPCon fall 2011 Fast and Good: Alternate Approaches to Quality at Etsy - STPCon fall 2011 Presentation Transcript

    • fast and goodAlternative Approaches to Quality at Etsy Noah Sussman STPCon Fall 2011, Dallas TX #STPCon
    • Our mission is to empower people to change the waythe global economy works. We see a world in whichvery-very small businesses have much-much moresway in shaping the economy, local living economiesare thriving everywhere, and people value authorshipand provenance as much as price and convenience. Weare bringing heart to commerce and making the worldmore fair, more sustainable, and more fun.
    • September 2011: 489 Web deployments 110 people committed to trunk4,519 changesets in all 9.2 changesets average per production deployment
    • September 2011: 489 Web deployments 80 people deployed to production20 or fewer committers in each deployment5.7 committers average per deployment
    • 10 million registered members25 million unique visitors per month800,000 active sellers150 different countries10 million listed items$314 million in sales in 2010
    • In 2010, MTTR was 6.4 minutesfor severity one outages
    • Six change-related outages in 2010Out of thousands of production deployments
    • “All existing revision control systems were built bypeople who build installed software.” ~Paul Hammond
    • All canonical software life cycles were designed bypeople who build installed software
    • “In the online world… there is one consumer of thesoftware: you. The users are consumers of theservice built atop the software.” ~ Theo Schlossnagle
    • Optimize for recoveryrather than focusing onpreventing failure.Failure is inevitable.
    • Default to open access
    • No blame
    • Use your ownjudgement
    • “an airport without an air traffic controller” Chad Dickerson
    • faster deploys = smaller changesets = fewer change-related outages
    • Self-serve deployment
    • princessbleeding edge code,production data and services
    • Shared development environment Single dev database Everyone commits to trunk
    • low-ceremony process
    • A test is a design document first
    • Many small deployments, many small tests
    • Beyond developer testing1. Test planning2. GUI-driven regression testing3. Performance testing4. Security / Penetration testing5. Nagios
    • Make it easy to write and run tests
    • Make it painless to use fixtures and mocks
    • Most bugs, most of the time, are easily nailed given evenan incomplete but suggestive characterization of theirerror conditions at source-code level. ~ESR, “How Many Eyeballs Tame Complexity,” 1997
    • Debugging is twice as hardas writing the code in thefirst place. Therefore, if youwrite the code as cleverly aspossible, you are, bydefinition, not smart enoughto debug it. ~ Brian Kernighan
    • Test to address technical debt
    • Keep the feedback loop short
    • Graphmonitorit didn’t happenAt Etsy we or over 40,000 real-time metrics
    • Business Metrics
    • Application Metrics
    • System Metrics
    • We hire our users
    • Tests are one of the tools we use todetect and prevent failure
    • There will always be bugs in production
    • Unexpected failures tend to be catastrophic
    • Anomaly detection is hard
    • Just ship
    • How do you deploy large features?
    • Releasing a feature is decoupled from deploying code
    • if ($cfg["new_search"]) { // new hotness $resp = search_solr();} else { // old busted $resp = search_grep();}
    • $cfg = array( checkout => true, homepage => true, profiles => true, new_search => false,);
    • Dark launchesAdmin-only rolloutsA/B are always testing
    • A culture of responsibility
    • A community of artisans
    • You are a participant observer
    • Noah
    • photo creditspenguins: paulmcdee photograph of snowy road: K. Leidorf in the mirror: Richard Avedon for the New YorkerWomen at work on C-47 Douglas cargo transport aircraft worker bike with wooden handlebars: Bertelli keys: Brian Cerveny office monitor wall and Etsy bike rack: Mashables Cubicle Spy fly: Lukjonis vs balrog: the brothers HildebrandtJust Ship t-shirt: Kellan raising: dogpose train coupling: David E. Smith I used your image and didnt credit you, its because I couldnt figure out who you were -- email me and I will gladly credit you.If you object to my using an image to which you own the rights, please contact me and I will remove that image from the deck.