Continuous improvement - Selenium Conf 2013

8,550 views
10,105 views

Published on

The past few years have seen a phenomenon of software organizations abandoning traditional release cycles in favor of daily or even hourly deployments. The emergence of a new, rapid software development workflow has raised questions regarding the role of test and QA in a product’s life cycle. When there is no QA phase, can there still be QA?

Published in: Technology, Business
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
8,550
On SlideShare
0
From Embeds
0
Number of Embeds
5,885
Actions
Shares
0
Downloads
0
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Continuous improvement - Selenium Conf 2013

  1. 1. ContinuousImprovementHowrapidreleasecyclesalterQAandtesting.NoahSussmanSeleniumConf2013@noahsussman#seconf
  2. 2. TheCanonicalAgileReleaseCycleSprintsoftwoweeksormoreinlength.Startdeploymentprocessattheendofthesprint.QAispartofthedeploymentprocess.QAmustbecompletebeforenewcodegoeslive.
  3. 3. TheContinuousDeliveryEnvironmentMinimumviablefeatureset.Deploymentisdecoupledfromrelease.Real-timedataonhowreleasesimpactrevenue.Constanttweakstolivefeatures.
  4. 4. HowContinousDeliveryDiffersFrom“Canonical”AgileProcessLargefeaturesaredeployedpiecemealovertimeConfigFlagsDarklaunchesWire-OffsEveryfeatureispartofanA/BcampaignThereisno“DoneDone”
  5. 5. Releasingafeatureisdecoupledfromdeployingcode.David E. Smith http://www.flickr.com/photos/david_e_smith/3566697122
  6. 6. Anairportwithoutanairtrafficcontroller.—ChadDickerson
  7. 7. ObservedBehaviorOfComplexSystemsEmergentbehaviorsrequireunplannedresponses.Improvements,tooarediscoverednotdesigned.Usersofthesystemhavecomplexexpectations.Suchsystemsarenever“complete.”
  8. 8. QAHappensWhen?Firstofall,whatis“QualityAssurance?”Authoritativelyassuringthattherearenodefects?No.Becausecompletetestingisimpossible.
  9. 9. Wearenotcustodiansofinherentlysafesystems.—SidneyDekker
  10. 10. Testingiseveryone’sjob.Library of Congress
  11. 11. FalsehoodsAboutQualityAssuranceThereareafinitenumberofbugs.Thereareafinitenumberofdetectablebugs.Allseverityonebugscanbefoundbeforerelease.Softwareisbuilttospecifications.Atsomepoint,softwareisfinished.
  12. 12. AndTheBiggestMythAboutQA…FALSE:Allbugshavelarge,complex,unpredictablecauses.Infact,manybugs—evenseverity1productionissues—resultfromerrorsthatatfirstglanceappeartrivial.
  13. 13. John Allspaw
  14. 14. ManySmallAnomaliesCombinedThe“SwissCheese”modelofriskpresentsuswithastrongcaseforprioritizingtheeliminationofsmallerrorsratherthanfocusingonthemitigationoflargecatastrophicfailures.Unittestingisgreatateliminatingsmallerrors.
  15. 15. Safetyisanemergentproperty…itserosionisnotaboutthebreakageorlackofqualityofsinglecomponents…managingqualityisaboutsinglecomponents,aboutseeinghowtheymeetparticularspecifications…—SidneyDekker
  16. 16. …thebestsoftwareengineeringprinciples…qualityassurance,testing…thehigheststandards—itsnotgoingtobeenough—NancyLeveson
  17. 17. Leveson’s3PracticesForLimitingRisk1.Oversight.2.Limitingcomplexity.3.Systemsthinking.
  18. 18. CharacteristicsofResilientSoftwareReadablecode.Reasonableautomatedtestcoverage.Sanearchitecture.Gooddebuggingtools.Anengineeringculturethatvaluesrefactoring.
  19. 19. QAHappensWhen???Exploratorytestingcanbeperformedanytime.Rigorous,scientificapproach.Focusoncustomersatisfactionratherthanaspec.Equallyusefulbeforeorafterarelease.
  20. 20. …exploratorytestingistestdesignandtestexecutionatthesametime.Thisistheoppositeofscriptedtesting(predefinedtestprocedures,whethermanualorautomated).Exploratorytests,unlikescriptedtests,arenotdefinedinadvanceandcarriedoutpreciselyaccordingtoplan.—JamesMarcusBach
  21. 21. ExploratoryTestingAddressesareasthatDeveloperTestingcan’t.Unit&Integrationtestsvalidateassumptions.There’salsoaneedtoinvalidateassumptions.
  22. 22. ManualTestingItdoesn’talwayslooklikeyouthinkitlooks.Real-TimeMonitoringisthenewfaceofmanualtesting.
  23. 23. Sufficientlyadvancedmonitoringisindistinguishablefromtesting.—EdKeyes
  24. 24. CaseStudy:WatchingTheGraphsAsof2012,Etsycollected>250,000real-timemetrics.Decidingwhichmetricsmatterisahumanproblem.Humanvisionexcelsatanomalydetection.Thus,atEtsy,everyonewatchesasubsetofthegraphs.
  25. 25. Anomalydetectionishard.
  26. 26. Mostbugs,mostofthetime,areeasilynailedgivenevenanincompletebutsuggestivecharacterizationoftheirerrorconditionsatsource-codelevel.—EricS.Raymond
  27. 27. Sourcecode,diffsandlogs.IfyourQAAnalystsdon’tlookatthese,teachthem!
  28. 28. Keepthefeedbackloopshort.
  29. 29. PayCloseAttentionToYourEffeciency-ToThoroughness-Trade-OffsContinuousreleasecycleshavedifferentrisksthanmonthlyorquarterlyreleasecycles.ButnoaspectofContinuousDeliveryislessrisky.
  30. 30. ThereIsNoSpecificationWatchyourgraphs.Testyourwholesystem.Incrementallyimproveyourproduct.
  31. 31. Questions?ns@noahsussman.com@noahsussmanfalsehoodsabouttime.com
  32. 32. FurtherReading“How Google Tests Software,” James Whittaker (especially chapter 5)“Look At Your Data,” John Rausser“Optimizing For Developer Happiness,” Chad Dickerson“Outages, Postmortems and Human Error,” John Allspaw“Prevention versus Governance versus Adaptive Capacities”, John AllspawThe Swiss Cheese Model of Risk (Wikipedia)The ETTO Principle (Wikipedia)“What Is Exploratory Testing?,” James Marcus Bach“How Many Eyeballs Tame Complexity,” Eric S. Raymond“The Timeless Way of Building,” Christopher Alexander“Ten Questions about Human Error”, Sidney Dekker“Sufficiently Advanced Monitoring is Indistinguishable From Testing”, Ed Keyes“SEC Technology Roundtable 10/2/2012”, Nancy Leveson“Reliable Software Systems Design,” Gerard J. Holzmann“The Power of Ten: Rules for Developing Safety Critical Code,” Gerard J. Holzmann

×