• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Continuous improvement - Selenium Conf 2013
 

Continuous improvement - Selenium Conf 2013

on

  • 2,183 views

The past few years have seen a phenomenon of software organizations abandoning traditional release cycles in favor of daily or even hourly deployments. The emergence of a new, rapid software ...

The past few years have seen a phenomenon of software organizations abandoning traditional release cycles in favor of daily or even hourly deployments. The emergence of a new, rapid software development workflow has raised questions regarding the role of test and QA in a product’s life cycle. When there is no QA phase, can there still be QA?

Statistics

Views

Total Views
2,183
Views on SlideShare
1,262
Embed Views
921

Actions

Likes
4
Downloads
0
Comments
0

9 Embeds 921

http://infiniteundo.com 868
https://twitter.com 22
http://cloud.feedly.com 22
http://gazeta.yandex.ru 2
http://newsblur.com 2
http://www.feedspot.com 2
http://reader.aol.com 1
http://translate.googleusercontent.com 1
http://www.goread.io 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Continuous improvement - Selenium Conf 2013 Continuous improvement - Selenium Conf 2013 Presentation Transcript

    • ContinuousImprovementHowrapidreleasecyclesalterQAandtesting.NoahSussmanSeleniumConf2013@noahsussman#seconf
    • TheCanonicalAgileReleaseCycleSprintsoftwoweeksormoreinlength.Startdeploymentprocessattheendofthesprint.QAispartofthedeploymentprocess.QAmustbecompletebeforenewcodegoeslive.
    • TheContinuousDeliveryEnvironmentMinimumviablefeatureset.Deploymentisdecoupledfromrelease.Real-timedataonhowreleasesimpactrevenue.Constanttweakstolivefeatures.
    • HowContinousDeliveryDiffersFrom“Canonical”AgileProcessLargefeaturesaredeployedpiecemealovertimeConfigFlagsDarklaunchesWire-OffsEveryfeatureispartofanA/BcampaignThereisno“DoneDone”
    • Releasingafeatureisdecoupledfromdeployingcode.David E. Smith http://www.flickr.com/photos/david_e_smith/3566697122
    • Anairportwithoutanairtrafficcontroller.—ChadDickerson
    • ObservedBehaviorOfComplexSystemsEmergentbehaviorsrequireunplannedresponses.Improvements,tooarediscoverednotdesigned.Usersofthesystemhavecomplexexpectations.Suchsystemsarenever“complete.”
    • QAHappensWhen?Firstofall,whatis“QualityAssurance?”Authoritativelyassuringthattherearenodefects?No.Becausecompletetestingisimpossible.
    • Wearenotcustodiansofinherentlysafesystems.—SidneyDekker
    • Testingiseveryone’sjob.Library of Congress
    • FalsehoodsAboutQualityAssuranceThereareafinitenumberofbugs.Thereareafinitenumberofdetectablebugs.Allseverityonebugscanbefoundbeforerelease.Softwareisbuilttospecifications.Atsomepoint,softwareisfinished.
    • AndTheBiggestMythAboutQA…FALSE:Allbugshavelarge,complex,unpredictablecauses.Infact,manybugs—evenseverity1productionissues—resultfromerrorsthatatfirstglanceappeartrivial.
    • John Allspaw
    • ManySmallAnomaliesCombinedThe“SwissCheese”modelofriskpresentsuswithastrongcaseforprioritizingtheeliminationofsmallerrorsratherthanfocusingonthemitigationoflargecatastrophicfailures.Unittestingisgreatateliminatingsmallerrors.
    • Safetyisanemergentproperty…itserosionisnotaboutthebreakageorlackofqualityofsinglecomponents…managingqualityisaboutsinglecomponents,aboutseeinghowtheymeetparticularspecifications…—SidneyDekker
    • …thebestsoftwareengineeringprinciples…qualityassurance,testing…thehigheststandards—itsnotgoingtobeenough—NancyLeveson
    • Leveson’s3PracticesForLimitingRisk1.Oversight.2.Limitingcomplexity.3.Systemsthinking.
    • CharacteristicsofResilientSoftwareReadablecode.Reasonableautomatedtestcoverage.Sanearchitecture.Gooddebuggingtools.Anengineeringculturethatvaluesrefactoring.
    • QAHappensWhen???Exploratorytestingcanbeperformedanytime.Rigorous,scientificapproach.Focusoncustomersatisfactionratherthanaspec.Equallyusefulbeforeorafterarelease.
    • …exploratorytestingistestdesignandtestexecutionatthesametime.Thisistheoppositeofscriptedtesting(predefinedtestprocedures,whethermanualorautomated).Exploratorytests,unlikescriptedtests,arenotdefinedinadvanceandcarriedoutpreciselyaccordingtoplan.—JamesMarcusBach
    • ExploratoryTestingAddressesareasthatDeveloperTestingcan’t.Unit&Integrationtestsvalidateassumptions.There’salsoaneedtoinvalidateassumptions.
    • ManualTestingItdoesn’talwayslooklikeyouthinkitlooks.Real-TimeMonitoringisthenewfaceofmanualtesting.
    • Sufficientlyadvancedmonitoringisindistinguishablefromtesting.—EdKeyes
    • CaseStudy:WatchingTheGraphsAsof2012,Etsycollected>250,000real-timemetrics.Decidingwhichmetricsmatterisahumanproblem.Humanvisionexcelsatanomalydetection.Thus,atEtsy,everyonewatchesasubsetofthegraphs.
    • Anomalydetectionishard.
    • Mostbugs,mostofthetime,areeasilynailedgivenevenanincompletebutsuggestivecharacterizationoftheirerrorconditionsatsource-codelevel.—EricS.Raymond
    • Sourcecode,diffsandlogs.IfyourQAAnalystsdon’tlookatthese,teachthem!
    • Keepthefeedbackloopshort.
    • PayCloseAttentionToYourEffeciency-ToThoroughness-Trade-OffsContinuousreleasecycleshavedifferentrisksthanmonthlyorquarterlyreleasecycles.ButnoaspectofContinuousDeliveryislessrisky.
    • ThereIsNoSpecificationWatchyourgraphs.Testyourwholesystem.Incrementallyimproveyourproduct.
    • Questions?ns@noahsussman.com@noahsussmanfalsehoodsabouttime.com
    • FurtherReading“How Google Tests Software,” James Whittaker (especially chapter 5)“Look At Your Data,” John Rausser“Optimizing For Developer Happiness,” Chad Dickerson“Outages, Postmortems and Human Error,” John Allspaw“Prevention versus Governance versus Adaptive Capacities”, John AllspawThe Swiss Cheese Model of Risk (Wikipedia)The ETTO Principle (Wikipedia)“What Is Exploratory Testing?,” James Marcus Bach“How Many Eyeballs Tame Complexity,” Eric S. Raymond“The Timeless Way of Building,” Christopher Alexander“Ten Questions about Human Error”, Sidney Dekker“Sufficiently Advanced Monitoring is Indistinguishable From Testing”, Ed Keyes“SEC Technology Roundtable 10/2/2012”, Nancy Leveson“Reliable Software Systems Design,” Gerard J. Holzmann“The Power of Ten: Rules for Developing Safety Critical Code,” Gerard J. Holzmann