Your SlideShare is downloading. ×
0
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Continuous improvement - Selenium Conf 2013
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Continuous improvement - Selenium Conf 2013

2,189

Published on

The past few years have seen a phenomenon of software organizations abandoning traditional release cycles in favor of daily or even hourly deployments. The emergence of a new, rapid software …

The past few years have seen a phenomenon of software organizations abandoning traditional release cycles in favor of daily or even hourly deployments. The emergence of a new, rapid software development workflow has raised questions regarding the role of test and QA in a product’s life cycle. When there is no QA phase, can there still be QA?

Published in: Technology, Business
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,189
On Slideshare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
0
Comments
0
Likes
4
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. ContinuousImprovementHowrapidreleasecyclesalterQAandtesting.NoahSussmanSeleniumConf2013@noahsussman#seconf
  • 2. TheCanonicalAgileReleaseCycleSprintsoftwoweeksormoreinlength.Startdeploymentprocessattheendofthesprint.QAispartofthedeploymentprocess.QAmustbecompletebeforenewcodegoeslive.
  • 3. TheContinuousDeliveryEnvironmentMinimumviablefeatureset.Deploymentisdecoupledfromrelease.Real-timedataonhowreleasesimpactrevenue.Constanttweakstolivefeatures.
  • 4. HowContinousDeliveryDiffersFrom“Canonical”AgileProcessLargefeaturesaredeployedpiecemealovertimeConfigFlagsDarklaunchesWire-OffsEveryfeatureispartofanA/BcampaignThereisno“DoneDone”
  • 5. Releasingafeatureisdecoupledfromdeployingcode.David E. Smith http://www.flickr.com/photos/david_e_smith/3566697122
  • 6. Anairportwithoutanairtrafficcontroller.—ChadDickerson
  • 7. ObservedBehaviorOfComplexSystemsEmergentbehaviorsrequireunplannedresponses.Improvements,tooarediscoverednotdesigned.Usersofthesystemhavecomplexexpectations.Suchsystemsarenever“complete.”
  • 8. QAHappensWhen?Firstofall,whatis“QualityAssurance?”Authoritativelyassuringthattherearenodefects?No.Becausecompletetestingisimpossible.
  • 9. Wearenotcustodiansofinherentlysafesystems.—SidneyDekker
  • 10. Testingiseveryone’sjob.Library of Congress
  • 11. FalsehoodsAboutQualityAssuranceThereareafinitenumberofbugs.Thereareafinitenumberofdetectablebugs.Allseverityonebugscanbefoundbeforerelease.Softwareisbuilttospecifications.Atsomepoint,softwareisfinished.
  • 12. AndTheBiggestMythAboutQA…FALSE:Allbugshavelarge,complex,unpredictablecauses.Infact,manybugs—evenseverity1productionissues—resultfromerrorsthatatfirstglanceappeartrivial.
  • 13. John Allspaw
  • 14. ManySmallAnomaliesCombinedThe“SwissCheese”modelofriskpresentsuswithastrongcaseforprioritizingtheeliminationofsmallerrorsratherthanfocusingonthemitigationoflargecatastrophicfailures.Unittestingisgreatateliminatingsmallerrors.
  • 15. Safetyisanemergentproperty…itserosionisnotaboutthebreakageorlackofqualityofsinglecomponents…managingqualityisaboutsinglecomponents,aboutseeinghowtheymeetparticularspecifications…—SidneyDekker
  • 16. …thebestsoftwareengineeringprinciples…qualityassurance,testing…thehigheststandards—itsnotgoingtobeenough—NancyLeveson
  • 17. Leveson’s3PracticesForLimitingRisk1.Oversight.2.Limitingcomplexity.3.Systemsthinking.
  • 18. CharacteristicsofResilientSoftwareReadablecode.Reasonableautomatedtestcoverage.Sanearchitecture.Gooddebuggingtools.Anengineeringculturethatvaluesrefactoring.
  • 19. QAHappensWhen???Exploratorytestingcanbeperformedanytime.Rigorous,scientificapproach.Focusoncustomersatisfactionratherthanaspec.Equallyusefulbeforeorafterarelease.
  • 20. …exploratorytestingistestdesignandtestexecutionatthesametime.Thisistheoppositeofscriptedtesting(predefinedtestprocedures,whethermanualorautomated).Exploratorytests,unlikescriptedtests,arenotdefinedinadvanceandcarriedoutpreciselyaccordingtoplan.—JamesMarcusBach
  • 21. ExploratoryTestingAddressesareasthatDeveloperTestingcan’t.Unit&Integrationtestsvalidateassumptions.There’salsoaneedtoinvalidateassumptions.
  • 22. ManualTestingItdoesn’talwayslooklikeyouthinkitlooks.Real-TimeMonitoringisthenewfaceofmanualtesting.
  • 23. Sufficientlyadvancedmonitoringisindistinguishablefromtesting.—EdKeyes
  • 24. CaseStudy:WatchingTheGraphsAsof2012,Etsycollected>250,000real-timemetrics.Decidingwhichmetricsmatterisahumanproblem.Humanvisionexcelsatanomalydetection.Thus,atEtsy,everyonewatchesasubsetofthegraphs.
  • 25. Anomalydetectionishard.
  • 26. Mostbugs,mostofthetime,areeasilynailedgivenevenanincompletebutsuggestivecharacterizationoftheirerrorconditionsatsource-codelevel.—EricS.Raymond
  • 27. Sourcecode,diffsandlogs.IfyourQAAnalystsdon’tlookatthese,teachthem!
  • 28. Keepthefeedbackloopshort.
  • 29. PayCloseAttentionToYourEffeciency-ToThoroughness-Trade-OffsContinuousreleasecycleshavedifferentrisksthanmonthlyorquarterlyreleasecycles.ButnoaspectofContinuousDeliveryislessrisky.
  • 30. ThereIsNoSpecificationWatchyourgraphs.Testyourwholesystem.Incrementallyimproveyourproduct.
  • 31. Questions?ns@noahsussman.com@noahsussmanfalsehoodsabouttime.com
  • 32. FurtherReading“How Google Tests Software,” James Whittaker (especially chapter 5)“Look At Your Data,” John Rausser“Optimizing For Developer Happiness,” Chad Dickerson“Outages, Postmortems and Human Error,” John Allspaw“Prevention versus Governance versus Adaptive Capacities”, John AllspawThe Swiss Cheese Model of Risk (Wikipedia)The ETTO Principle (Wikipedia)“What Is Exploratory Testing?,” James Marcus Bach“How Many Eyeballs Tame Complexity,” Eric S. Raymond“The Timeless Way of Building,” Christopher Alexander“Ten Questions about Human Error”, Sidney Dekker“Sufficiently Advanced Monitoring is Indistinguishable From Testing”, Ed Keyes“SEC Technology Roundtable 10/2/2012”, Nancy Leveson“Reliable Software Systems Design,” Gerard J. Holzmann“The Power of Ten: Rules for Developing Safety Critical Code,” Gerard J. Holzmann

×