Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Fast and Good: Alternate Approaches to Quality at Etsy - STPCon fall 2011


Published on

Published in: Technology, Sports
  • Dating direct: ❶❶❶ ❶❶❶
    Are you sure you want to  Yes  No
    Your message goes here
  • Dating for everyone is here: ♥♥♥ ♥♥♥
    Are you sure you want to  Yes  No
    Your message goes here
  • Here are a couple of brief video clips where I summarize my talk. (thanks uTest!) and (thanks Search Software Quality!)
    Are you sure you want to  Yes  No
    Your message goes here
  • At Etsy we have spent the last several years learning to move fast at scale. Building on the precedents set by organizations like Flickr, IMVU and Facebook, we have created a testing methodology that is highly automated, low-ceremony and involves participation from individuals across the organization. As our development cycle became more rapid many challenges to testing arose: What is the exploratory tester’s role in such a fast-moving environment? How does one validate simultaneous releases by many different functional teams? How would the feedback loop between the tester and designer need to change? Many of the answers to these questions appeared counter-intuitive at first: never cutting release branches, eliminating the staging environment, asking engineers to assume responsibility for testing their own code. The end result is an online marketplace with hundreds of thousands of active sellers collectively drawing over a billion page views a month. This is the story of how the Etsy team learned to move fast, mitigate risk and deliver quality.
    Are you sure you want to  Yes  No
    Your message goes here

Fast and Good: Alternate Approaches to Quality at Etsy - STPCon fall 2011

  1. 1. fast and goodAlternative Approaches to Quality at Etsy Noah Sussman STPCon Fall 2011, Dallas TX #STPCon
  2. 2. Our mission is to empower people to change the waythe global economy works. We see a world in whichvery-very small businesses have much-much moresway in shaping the economy, local living economiesare thriving everywhere, and people value authorshipand provenance as much as price and convenience. Weare bringing heart to commerce and making the worldmore fair, more sustainable, and more fun.
  3. 3. September 2011: 489 Web deployments 110 people committed to trunk4,519 changesets in all 9.2 changesets average per production deployment
  4. 4. September 2011: 489 Web deployments 80 people deployed to production20 or fewer committers in each deployment5.7 committers average per deployment
  5. 5. 10 million registered members25 million unique visitors per month800,000 active sellers150 different countries10 million listed items$314 million in sales in 2010
  6. 6. In 2010, MTTR was 6.4 minutesfor severity one outages
  7. 7. Six change-related outages in 2010Out of thousands of production deployments
  8. 8. “All existing revision control systems were built bypeople who build installed software.” ~Paul Hammond
  9. 9. All canonical software life cycles were designed bypeople who build installed software
  10. 10. “In the online world… there is one consumer of thesoftware: you. The users are consumers of theservice built atop the software.” ~ Theo Schlossnagle
  11. 11. Optimize for recoveryrather than focusing onpreventing failure.Failure is inevitable.
  12. 12. Default to open access
  13. 13. No blame
  14. 14. Use your ownjudgement
  15. 15. “an airport without an air traffic controller” Chad Dickerson
  16. 16. faster deploys = smaller changesets = fewer change-related outages
  17. 17. Self-serve deployment
  18. 18. princessbleeding edge code,production data and services
  19. 19. Shared development environment Single dev database Everyone commits to trunk
  20. 20. low-ceremony process
  21. 21. A test is a design document first
  22. 22. Many small deployments, many small tests
  23. 23. Beyond developer testing1. Test planning2. GUI-driven regression testing3. Performance testing4. Security / Penetration testing5. Nagios
  24. 24. Make it easy to write and run tests
  25. 25. Make it painless to use fixtures and mocks
  26. 26. Most bugs, most of the time, are easily nailed given evenan incomplete but suggestive characterization of theirerror conditions at source-code level. ~ESR, “How Many Eyeballs Tame Complexity,” 1997
  27. 27. Debugging is twice as hardas writing the code in thefirst place. Therefore, if youwrite the code as cleverly aspossible, you are, bydefinition, not smart enoughto debug it. ~ Brian Kernighan
  28. 28. Test to address technical debt
  29. 29. Keep the feedback loop short
  30. 30. Graphmonitorit didn’t happenAt Etsy we or over 40,000 real-time metrics
  31. 31. Business Metrics
  32. 32. Application Metrics
  33. 33. System Metrics
  34. 34. We hire our users
  35. 35. Tests are one of the tools we use todetect and prevent failure
  36. 36. There will always be bugs in production
  37. 37. Unexpected failures tend to be catastrophic
  38. 38. Anomaly detection is hard
  39. 39. Just ship
  40. 40. How do you deploy large features?
  41. 41. Releasing a feature is decoupled from deploying code
  42. 42. if ($cfg["new_search"]) { // new hotness $resp = search_solr();} else { // old busted $resp = search_grep();}
  43. 43. $cfg = array( checkout => true, homepage => true, profiles => true, new_search => false,);
  44. 44. Dark launchesAdmin-only rolloutsA/B are always testing
  45. 45. A culture of responsibility
  46. 46. A community of artisans
  47. 47. You are a participant observer
  48. 48.
  49. 49. Noah
  50. 50. photo creditspenguins: paulmcdee photograph of snowy road: K. Leidorf in the mirror: Richard Avedon for the New YorkerWomen at work on C-47 Douglas cargo transport aircraft worker bike with wooden handlebars: Bertelli keys: Brian Cerveny office monitor wall and Etsy bike rack: Mashables Cubicle Spy fly: Lukjonis vs balrog: the brothers HildebrandtJust Ship t-shirt: Kellan raising: dogpose train coupling: David E. Smith I used your image and didnt credit you, its because I couldnt figure out who you were -- email me and I will gladly credit you.If you object to my using an image to which you own the rights, please contact me and I will remove that image from the deck.