ContinuousImprovement  How rapid release cycles    alter QA and testing.       Noah SussmanDevopsDay Mountain View, 2012  ...
The Canonical Agile    Release CycleSprints of two weeks or more in length.Start deployment process at the end of the spri...
The Continuous       Release CycleMinimum viable feature set.Deployment is decoupled from release.Real-time data on how re...
Releasing a featureis decoupled fromdeploying code.    David E. Smith http://www.flickr.com/photos/david_e_smith/3566697122
An airport withoutan air traffic controller.         —Chad Dickerson
This Part Really IsDifferent From AgileLarge features are deployed piecemeal over time.Every feature is part of an A/B cam...
Observed BehaviorOf Complex SystemsEmergent behaviors require unplanned responses.Improvements, too are discovered not des...
QA Happens When?First of all, what is “Quality Assurance?”Authoritatively assuring that there are no defects?That’s imposs...
Testing is everyone’s job.
Myths About Bug       DetectionThere are a finite number of bugs.There are a finite number of detectable bugs.All severity o...
The Biggest MythBugs have complex, unpredictable causes.In fact, most errors in software are the results ofincorrect assum...
Many SmallAnomalies CombinedThe “Swiss Cheese” model of risk presents uswith a strong case for prioritizing the eliminatio...
The whole time I’mprogramming, I’mconstantly checkingmy assumptions.   —Rasmus Lerdorf
Resilience, Not          “Quality”Readable code.Reasonable test coverage.Sane architecture.Good debugging tools.An enginee...
Manual TestingIt doesn’t always look like you think it looks.Real-Time Monitoring is the new face of testing.
Anomaly detection is hard.
Watching The GraphsEtsy collects well over a quarter million metrics.Deciding which ones matter is a human problem.Everyon...
QA Happens When??Exploratory testing can be performed any time.Rigorous, scientific approach.Focus on customer satisfaction...
Just Quality“Assurance” is a terrible word. Let’s discard it.Quality exists, we just can’t assure or prove it.
There Is No Such  Thing As A Formal   Proof Of Quality.Yet most of us would agree it exists.I propose that “customer exper...
Exploratory TestingAddresses areas that Developer Testing can’t.Developer Testing validates assumptions.The Tester’s job i...
Technology InformsCustomer ExperienceExploratory Testing requires an understanding ofthe ways in which a whole system is i...
Most bugs, most of thetime, are easily nailed giveneven an incomplete butsuggestive characterizationof their error conditi...
Source, diffs, logs.If your QA Analysts don’t look at these, teach them.
Customer SupportYour customer support operators spend more timetalking to your users than anyone else.Customer Support int...
Keep the feedback loop short.
Manage Your Culture.
Effeciency To         Thoroughness           Trade-OffRapid release cycles have different risks thanslower release cycles....
Test EverywhereForeseeable errors can be worked out in dev.Unforeseeable errors must be worked out in prod.
Fail ForwardLet go of the idea of “last stable release.”Software exists in context.Networks, services and people are alway...
Forget About        Satisfying The        RequirementsWatch your graphs.Listen to your customers.Adhere to your protocols....
Questions?@noahsussmanns@noahsussman.cominfiniteundo.com
Further Reading“How Google Tests Software,” James Whittaker (especially chapter 5)“Look At Your Data,” John Rausser“Optimi...
Continuous improvement   devops day mountain view 2012
Continuous improvement   devops day mountain view 2012
Continuous improvement   devops day mountain view 2012
Continuous improvement   devops day mountain view 2012
Continuous improvement   devops day mountain view 2012
Continuous improvement   devops day mountain view 2012
Upcoming SlideShare
Loading in …5
×

Continuous improvement devops day mountain view 2012

3,677 views
3,526 views

Published on

The past few years have seen a phenomenon of software organizations abandoning traditional release cycles in favor of daily or even hourly deployments. The emergence of a new, rapid software development workflow has raised questions regarding the role of test and QA in a product's life cycle. When there is no QA phase, can there still be QA?

Indeed, as the speed of product development changes, risk must be assessed differently. A monthly release cycle provides time to assure a product is reasonably free of defects. In a rapid release cycle, such assurance is impossible. Quality instead comes to mean assuring that the product is capable of recovering from defects.

I will discuss how software risk mitigation practice changes as the speed of development increases. And I will explore the idea that recovering from failure is a far more pragmatic goal than preventing failure in the first place.

Published in: Technology

Continuous improvement devops day mountain view 2012

  1. 1. ContinuousImprovement How rapid release cycles alter QA and testing. Noah SussmanDevopsDay Mountain View, 2012 @noahsussman #devopsdays
  2. 2. The Canonical Agile Release CycleSprints of two weeks or more in length.Start deployment process at the end of the sprint.QA is part of the deployment process.QA must be complete before new code goes live.
  3. 3. The Continuous Release CycleMinimum viable feature set.Deployment is decoupled from release.Real-time data on how releases impact revenue.Constant tweaks to live features.
  4. 4. Releasing a featureis decoupled fromdeploying code. David E. Smith http://www.flickr.com/photos/david_e_smith/3566697122
  5. 5. An airport withoutan air traffic controller. —Chad Dickerson
  6. 6. This Part Really IsDifferent From AgileLarge features are deployed piecemeal over time.Every feature is part of an A/B campaign.Dark and Admin-Only launches.Wire-Offs and Config Flags.There is no “Done Done.”
  7. 7. Observed BehaviorOf Complex SystemsEmergent behaviors require unplanned responses.Improvements, too are discovered not designed.Users of the system have complex expectations.Such systems are never “complete.”
  8. 8. QA Happens When?First of all, what is “Quality Assurance?”Authoritatively assuring that there are no defects?That’s impossible.
  9. 9. Testing is everyone’s job.
  10. 10. Myths About Bug DetectionThere are a finite number of bugs.There are a finite number of detectable bugs.All severity one bugs can be found before releaseSoftware is built to specifications.At some point, software is finished.
  11. 11. The Biggest MythBugs have complex, unpredictable causes.In fact, most errors in software are the results ofincorrect assumptions made by programmers.
  12. 12. Many SmallAnomalies CombinedThe “Swiss Cheese” model of risk presents uswith a strong case for prioritizing the elimination ofsmall errors rather than focusing on the mitigationof large catastrophic failures.Unit testing is great at eliminating small errors.
  13. 13. The whole time I’mprogramming, I’mconstantly checkingmy assumptions. —Rasmus Lerdorf
  14. 14. Resilience, Not “Quality”Readable code.Reasonable test coverage.Sane architecture.Good debugging tools.An engineering culture that values refactoring.These are measureable goals.
  15. 15. Manual TestingIt doesn’t always look like you think it looks.Real-Time Monitoring is the new face of testing.
  16. 16. Anomaly detection is hard.
  17. 17. Watching The GraphsEtsy collects well over a quarter million metrics.Deciding which ones matter is a human problem.Everyone watches some subset of the graphs.Human vision excels at anomaly detection.
  18. 18. QA Happens When??Exploratory testing can be performed any time.Rigorous, scientific approach.Focus on customer satisfaction rather than a spec.Equally useful before or after a release.
  19. 19. Just Quality“Assurance” is a terrible word. Let’s discard it.Quality exists, we just can’t assure or prove it.
  20. 20. There Is No Such Thing As A Formal Proof Of Quality.Yet most of us would agree it exists.I propose that “customer experience” is a betterterm-of-art-than “quality.”Though there’s no formal proof for that either.
  21. 21. Exploratory TestingAddresses areas that Developer Testing can’t.Developer Testing validates assumptions.The Tester’s job is to invalidate assumptions.
  22. 22. Technology InformsCustomer ExperienceExploratory Testing requires an understanding ofthe ways in which a whole system is intended toserve a community of users.The problem space has as much to do withtechnology as it does with product requirements.
  23. 23. Most bugs, most of thetime, are easily nailed giveneven an incomplete butsuggestive characterizationof their error conditions atsource-code level. —Eric S. Raymond
  24. 24. Source, diffs, logs.If your QA Analysts don’t look at these, teach them.
  25. 25. Customer SupportYour customer support operators spend more timetalking to your users than anyone else.Customer Support interface with users asindividuals rather than as aggregate data.
  26. 26. Keep the feedback loop short.
  27. 27. Manage Your Culture.
  28. 28. Effeciency To Thoroughness Trade-OffRapid release cycles have different risks thanslower release cycles.But nothing about risk itself has changed.
  29. 29. Test EverywhereForeseeable errors can be worked out in dev.Unforeseeable errors must be worked out in prod.
  30. 30. Fail ForwardLet go of the idea of “last stable release.”Software exists in context.Networks, services and people are always in flux.
  31. 31. Forget About Satisfying The RequirementsWatch your graphs.Listen to your customers.Adhere to your protocols.Improve your product.
  32. 32. Questions?@noahsussmanns@noahsussman.cominfiniteundo.com
  33. 33. Further Reading“How Google Tests Software,” James Whittaker (especially chapter 5)“Look At Your Data,” John Rausser“Optimizing For Developer Happiness,” Chad Dickerson“Outages, Postmortems and Human Error,” John Allspawhttp://en.wikipedia.org/wiki/Swiss_cheese_model“What Is Exploratory Testing?,” James Bach“How Many Eyeballs Tame Complexity,” ESR“The Timeless Way of Building,” Christopher Alexander

×