ContinuousImprovementHowrapidreleasecyclesalterQAandtesting.NoahSussmanSeleniumConf2013@noahsussman#seconf
TheCanonicalAgileReleaseCycleSprintsoftwoweeksormoreinlength.Startdeploymentprocessattheendofthesprint.QAispartofthedeploy...
TheContinuousDeliveryEnvironmentMinimumviablefeatureset.Deploymentisdecoupledfromrelease.Real-timedataonhowreleasesimpactr...
HowContinousDeliveryDiffersFrom“Canonical”AgileProcessLargefeaturesaredeployedpiecemealovertimeConfigFlagsDarklaunchesWire...
Releasingafeatureisdecoupledfromdeployingcode.David E. Smith http://www.flickr.com/photos/david_e_smith/3566697122
Anairportwithoutanairtrafficcontroller.—ChadDickerson
ObservedBehaviorOfComplexSystemsEmergentbehaviorsrequireunplannedresponses.Improvements,tooarediscoverednotdesigned.Userso...
QAHappensWhen?Firstofall,whatis“QualityAssurance?”Authoritativelyassuringthattherearenodefects?No.Becausecompletetestingis...
Wearenotcustodiansofinherentlysafesystems.—SidneyDekker
Testingiseveryone’sjob.Library of Congress
FalsehoodsAboutQualityAssuranceThereareafinitenumberofbugs.Thereareafinitenumberofdetectablebugs.Allseverityonebugscanbefo...
AndTheBiggestMythAboutQA…FALSE:Allbugshavelarge,complex,unpredictablecauses.Infact,manybugs—evenseverity1productionissues—...
John Allspaw
ManySmallAnomaliesCombinedThe“SwissCheese”modelofriskpresentsuswithastrongcaseforprioritizingtheeliminationofsmallerrorsra...
Safetyisanemergentproperty…itserosionisnotaboutthebreakageorlackofqualityofsinglecomponents…managingqualityisaboutsingleco...
…thebestsoftwareengineeringprinciples…qualityassurance,testing…thehigheststandards—itsnotgoingtobeenough—NancyLeveson
Leveson’s3PracticesForLimitingRisk1.Oversight.2.Limitingcomplexity.3.Systemsthinking.
CharacteristicsofResilientSoftwareReadablecode.Reasonableautomatedtestcoverage.Sanearchitecture.Gooddebuggingtools.Anengin...
QAHappensWhen???Exploratorytestingcanbeperformedanytime.Rigorous,scientificapproach.Focusoncustomersatisfactionratherthana...
…exploratorytestingistestdesignandtestexecutionatthesametime.Thisistheoppositeofscriptedtesting(predefinedtestprocedures,w...
ExploratoryTestingAddressesareasthatDeveloperTestingcan’t.Unit&Integrationtestsvalidateassumptions.There’salsoaneedtoinval...
ManualTestingItdoesn’talwayslooklikeyouthinkitlooks.Real-TimeMonitoringisthenewfaceofmanualtesting.
Sufficientlyadvancedmonitoringisindistinguishablefromtesting.—EdKeyes
CaseStudy:WatchingTheGraphsAsof2012,Etsycollected>250,000real-timemetrics.Decidingwhichmetricsmatterisahumanproblem.Humanv...
Anomalydetectionishard.
Mostbugs,mostofthetime,areeasilynailedgivenevenanincompletebutsuggestivecharacterizationoftheirerrorconditionsatsource-cod...
Sourcecode,diffsandlogs.IfyourQAAnalystsdon’tlookatthese,teachthem!
Keepthefeedbackloopshort.
PayCloseAttentionToYourEffeciency-ToThoroughness-Trade-OffsContinuousreleasecycleshavedifferentrisksthanmonthlyorquarterly...
ThereIsNoSpecificationWatchyourgraphs.Testyourwholesystem.Incrementallyimproveyourproduct.
Questions?ns@noahsussman.com@noahsussmanfalsehoodsabouttime.com
FurtherReading“How Google Tests Software,” James Whittaker (especially chapter 5)“Look At Your Data,” John Rausser“Optimiz...
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Upcoming SlideShare
Loading in...5
×

Continuous improvement - Selenium Conf 2013

2,287

Published on

The past few years have seen a phenomenon of software organizations abandoning traditional release cycles in favor of daily or even hourly deployments. The emergence of a new, rapid software development workflow has raised questions regarding the role of test and QA in a product’s life cycle. When there is no QA phase, can there still be QA?

Published in: Technology, Business
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,287
On Slideshare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
0
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Continuous improvement - Selenium Conf 2013

  1. 1. ContinuousImprovementHowrapidreleasecyclesalterQAandtesting.NoahSussmanSeleniumConf2013@noahsussman#seconf
  2. 2. TheCanonicalAgileReleaseCycleSprintsoftwoweeksormoreinlength.Startdeploymentprocessattheendofthesprint.QAispartofthedeploymentprocess.QAmustbecompletebeforenewcodegoeslive.
  3. 3. TheContinuousDeliveryEnvironmentMinimumviablefeatureset.Deploymentisdecoupledfromrelease.Real-timedataonhowreleasesimpactrevenue.Constanttweakstolivefeatures.
  4. 4. HowContinousDeliveryDiffersFrom“Canonical”AgileProcessLargefeaturesaredeployedpiecemealovertimeConfigFlagsDarklaunchesWire-OffsEveryfeatureispartofanA/BcampaignThereisno“DoneDone”
  5. 5. Releasingafeatureisdecoupledfromdeployingcode.David E. Smith http://www.flickr.com/photos/david_e_smith/3566697122
  6. 6. Anairportwithoutanairtrafficcontroller.—ChadDickerson
  7. 7. ObservedBehaviorOfComplexSystemsEmergentbehaviorsrequireunplannedresponses.Improvements,tooarediscoverednotdesigned.Usersofthesystemhavecomplexexpectations.Suchsystemsarenever“complete.”
  8. 8. QAHappensWhen?Firstofall,whatis“QualityAssurance?”Authoritativelyassuringthattherearenodefects?No.Becausecompletetestingisimpossible.
  9. 9. Wearenotcustodiansofinherentlysafesystems.—SidneyDekker
  10. 10. Testingiseveryone’sjob.Library of Congress
  11. 11. FalsehoodsAboutQualityAssuranceThereareafinitenumberofbugs.Thereareafinitenumberofdetectablebugs.Allseverityonebugscanbefoundbeforerelease.Softwareisbuilttospecifications.Atsomepoint,softwareisfinished.
  12. 12. AndTheBiggestMythAboutQA…FALSE:Allbugshavelarge,complex,unpredictablecauses.Infact,manybugs—evenseverity1productionissues—resultfromerrorsthatatfirstglanceappeartrivial.
  13. 13. John Allspaw
  14. 14. ManySmallAnomaliesCombinedThe“SwissCheese”modelofriskpresentsuswithastrongcaseforprioritizingtheeliminationofsmallerrorsratherthanfocusingonthemitigationoflargecatastrophicfailures.Unittestingisgreatateliminatingsmallerrors.
  15. 15. Safetyisanemergentproperty…itserosionisnotaboutthebreakageorlackofqualityofsinglecomponents…managingqualityisaboutsinglecomponents,aboutseeinghowtheymeetparticularspecifications…—SidneyDekker
  16. 16. …thebestsoftwareengineeringprinciples…qualityassurance,testing…thehigheststandards—itsnotgoingtobeenough—NancyLeveson
  17. 17. Leveson’s3PracticesForLimitingRisk1.Oversight.2.Limitingcomplexity.3.Systemsthinking.
  18. 18. CharacteristicsofResilientSoftwareReadablecode.Reasonableautomatedtestcoverage.Sanearchitecture.Gooddebuggingtools.Anengineeringculturethatvaluesrefactoring.
  19. 19. QAHappensWhen???Exploratorytestingcanbeperformedanytime.Rigorous,scientificapproach.Focusoncustomersatisfactionratherthanaspec.Equallyusefulbeforeorafterarelease.
  20. 20. …exploratorytestingistestdesignandtestexecutionatthesametime.Thisistheoppositeofscriptedtesting(predefinedtestprocedures,whethermanualorautomated).Exploratorytests,unlikescriptedtests,arenotdefinedinadvanceandcarriedoutpreciselyaccordingtoplan.—JamesMarcusBach
  21. 21. ExploratoryTestingAddressesareasthatDeveloperTestingcan’t.Unit&Integrationtestsvalidateassumptions.There’salsoaneedtoinvalidateassumptions.
  22. 22. ManualTestingItdoesn’talwayslooklikeyouthinkitlooks.Real-TimeMonitoringisthenewfaceofmanualtesting.
  23. 23. Sufficientlyadvancedmonitoringisindistinguishablefromtesting.—EdKeyes
  24. 24. CaseStudy:WatchingTheGraphsAsof2012,Etsycollected>250,000real-timemetrics.Decidingwhichmetricsmatterisahumanproblem.Humanvisionexcelsatanomalydetection.Thus,atEtsy,everyonewatchesasubsetofthegraphs.
  25. 25. Anomalydetectionishard.
  26. 26. Mostbugs,mostofthetime,areeasilynailedgivenevenanincompletebutsuggestivecharacterizationoftheirerrorconditionsatsource-codelevel.—EricS.Raymond
  27. 27. Sourcecode,diffsandlogs.IfyourQAAnalystsdon’tlookatthese,teachthem!
  28. 28. Keepthefeedbackloopshort.
  29. 29. PayCloseAttentionToYourEffeciency-ToThoroughness-Trade-OffsContinuousreleasecycleshavedifferentrisksthanmonthlyorquarterlyreleasecycles.ButnoaspectofContinuousDeliveryislessrisky.
  30. 30. ThereIsNoSpecificationWatchyourgraphs.Testyourwholesystem.Incrementallyimproveyourproduct.
  31. 31. Questions?ns@noahsussman.com@noahsussmanfalsehoodsabouttime.com
  32. 32. FurtherReading“How Google Tests Software,” James Whittaker (especially chapter 5)“Look At Your Data,” John Rausser“Optimizing For Developer Happiness,” Chad Dickerson“Outages, Postmortems and Human Error,” John Allspaw“Prevention versus Governance versus Adaptive Capacities”, John AllspawThe Swiss Cheese Model of Risk (Wikipedia)The ETTO Principle (Wikipedia)“What Is Exploratory Testing?,” James Marcus Bach“How Many Eyeballs Tame Complexity,” Eric S. Raymond“The Timeless Way of Building,” Christopher Alexander“Ten Questions about Human Error”, Sidney Dekker“Sufficiently Advanced Monitoring is Indistinguishable From Testing”, Ed Keyes“SEC Technology Roundtable 10/2/2012”, Nancy Leveson“Reliable Software Systems Design,” Gerard J. Holzmann“The Power of Ten: Rules for Developing Safety Critical Code,” Gerard J. Holzmann

×