• Save
Continuous improvement - Selenium Conf 2013
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Continuous improvement - Selenium Conf 2013

  • 2,375 views
Uploaded on

The past few years have seen a phenomenon of software organizations abandoning traditional release cycles in favor of daily or even hourly deployments. The emergence of a new, rapid software......

The past few years have seen a phenomenon of software organizations abandoning traditional release cycles in favor of daily or even hourly deployments. The emergence of a new, rapid software development workflow has raised questions regarding the role of test and QA in a product’s life cycle. When there is no QA phase, can there still be QA?

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
2,375
On Slideshare
1,429
From Embeds
946
Number of Embeds
9

Actions

Shares
Downloads
0
Comments
0
Likes
4

Embeds 946

http://infiniteundo.com 893
https://twitter.com 22
http://cloud.feedly.com 22
http://gazeta.yandex.ru 2
http://newsblur.com 2
http://www.feedspot.com 2
http://reader.aol.com 1
http://translate.googleusercontent.com 1
http://www.goread.io 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. ContinuousImprovementHowrapidreleasecyclesalterQAandtesting.NoahSussmanSeleniumConf2013@noahsussman#seconf
  • 2. TheCanonicalAgileReleaseCycleSprintsoftwoweeksormoreinlength.Startdeploymentprocessattheendofthesprint.QAispartofthedeploymentprocess.QAmustbecompletebeforenewcodegoeslive.
  • 3. TheContinuousDeliveryEnvironmentMinimumviablefeatureset.Deploymentisdecoupledfromrelease.Real-timedataonhowreleasesimpactrevenue.Constanttweakstolivefeatures.
  • 4. HowContinousDeliveryDiffersFrom“Canonical”AgileProcessLargefeaturesaredeployedpiecemealovertimeConfigFlagsDarklaunchesWire-OffsEveryfeatureispartofanA/BcampaignThereisno“DoneDone”
  • 5. Releasingafeatureisdecoupledfromdeployingcode.David E. Smith http://www.flickr.com/photos/david_e_smith/3566697122
  • 6. Anairportwithoutanairtrafficcontroller.—ChadDickerson
  • 7. ObservedBehaviorOfComplexSystemsEmergentbehaviorsrequireunplannedresponses.Improvements,tooarediscoverednotdesigned.Usersofthesystemhavecomplexexpectations.Suchsystemsarenever“complete.”
  • 8. QAHappensWhen?Firstofall,whatis“QualityAssurance?”Authoritativelyassuringthattherearenodefects?No.Becausecompletetestingisimpossible.
  • 9. Wearenotcustodiansofinherentlysafesystems.—SidneyDekker
  • 10. Testingiseveryone’sjob.Library of Congress
  • 11. FalsehoodsAboutQualityAssuranceThereareafinitenumberofbugs.Thereareafinitenumberofdetectablebugs.Allseverityonebugscanbefoundbeforerelease.Softwareisbuilttospecifications.Atsomepoint,softwareisfinished.
  • 12. AndTheBiggestMythAboutQA…FALSE:Allbugshavelarge,complex,unpredictablecauses.Infact,manybugs—evenseverity1productionissues—resultfromerrorsthatatfirstglanceappeartrivial.
  • 13. John Allspaw
  • 14. ManySmallAnomaliesCombinedThe“SwissCheese”modelofriskpresentsuswithastrongcaseforprioritizingtheeliminationofsmallerrorsratherthanfocusingonthemitigationoflargecatastrophicfailures.Unittestingisgreatateliminatingsmallerrors.
  • 15. Safetyisanemergentproperty…itserosionisnotaboutthebreakageorlackofqualityofsinglecomponents…managingqualityisaboutsinglecomponents,aboutseeinghowtheymeetparticularspecifications…—SidneyDekker
  • 16. …thebestsoftwareengineeringprinciples…qualityassurance,testing…thehigheststandards—itsnotgoingtobeenough—NancyLeveson
  • 17. Leveson’s3PracticesForLimitingRisk1.Oversight.2.Limitingcomplexity.3.Systemsthinking.
  • 18. CharacteristicsofResilientSoftwareReadablecode.Reasonableautomatedtestcoverage.Sanearchitecture.Gooddebuggingtools.Anengineeringculturethatvaluesrefactoring.
  • 19. QAHappensWhen???Exploratorytestingcanbeperformedanytime.Rigorous,scientificapproach.Focusoncustomersatisfactionratherthanaspec.Equallyusefulbeforeorafterarelease.
  • 20. …exploratorytestingistestdesignandtestexecutionatthesametime.Thisistheoppositeofscriptedtesting(predefinedtestprocedures,whethermanualorautomated).Exploratorytests,unlikescriptedtests,arenotdefinedinadvanceandcarriedoutpreciselyaccordingtoplan.—JamesMarcusBach
  • 21. ExploratoryTestingAddressesareasthatDeveloperTestingcan’t.Unit&Integrationtestsvalidateassumptions.There’salsoaneedtoinvalidateassumptions.
  • 22. ManualTestingItdoesn’talwayslooklikeyouthinkitlooks.Real-TimeMonitoringisthenewfaceofmanualtesting.
  • 23. Sufficientlyadvancedmonitoringisindistinguishablefromtesting.—EdKeyes
  • 24. CaseStudy:WatchingTheGraphsAsof2012,Etsycollected>250,000real-timemetrics.Decidingwhichmetricsmatterisahumanproblem.Humanvisionexcelsatanomalydetection.Thus,atEtsy,everyonewatchesasubsetofthegraphs.
  • 25. Anomalydetectionishard.
  • 26. Mostbugs,mostofthetime,areeasilynailedgivenevenanincompletebutsuggestivecharacterizationoftheirerrorconditionsatsource-codelevel.—EricS.Raymond
  • 27. Sourcecode,diffsandlogs.IfyourQAAnalystsdon’tlookatthese,teachthem!
  • 28. Keepthefeedbackloopshort.
  • 29. PayCloseAttentionToYourEffeciency-ToThoroughness-Trade-OffsContinuousreleasecycleshavedifferentrisksthanmonthlyorquarterlyreleasecycles.ButnoaspectofContinuousDeliveryislessrisky.
  • 30. ThereIsNoSpecificationWatchyourgraphs.Testyourwholesystem.Incrementallyimproveyourproduct.
  • 31. Questions?ns@noahsussman.com@noahsussmanfalsehoodsabouttime.com
  • 32. FurtherReading“How Google Tests Software,” James Whittaker (especially chapter 5)“Look At Your Data,” John Rausser“Optimizing For Developer Happiness,” Chad Dickerson“Outages, Postmortems and Human Error,” John Allspaw“Prevention versus Governance versus Adaptive Capacities”, John AllspawThe Swiss Cheese Model of Risk (Wikipedia)The ETTO Principle (Wikipedia)“What Is Exploratory Testing?,” James Marcus Bach“How Many Eyeballs Tame Complexity,” Eric S. Raymond“The Timeless Way of Building,” Christopher Alexander“Ten Questions about Human Error”, Sidney Dekker“Sufficiently Advanced Monitoring is Indistinguishable From Testing”, Ed Keyes“SEC Technology Roundtable 10/2/2012”, Nancy Leveson“Reliable Software Systems Design,” Gerard J. Holzmann“The Power of Ten: Rules for Developing Safety Critical Code,” Gerard J. Holzmann