Test automation Anecdotes


Published on

Dicas de automação de teste de software

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Test automation Anecdotes

  1. 1. EuroSTARSoftware TestingC o n fe r e n c eEuroSTARSoftware TestingC o m m u n i t yTest AutomationAnecdotesDorothy Graham,Independent Consultant, UK &Mark Fewster,Grove Consultants, UK
  2. 2. Test Automation Anecdotes1PAGE IntroductionSoftware test automation has moved beyonda luxury to become a necessity. Applicationsand systems have grown ever larger and morecomplex, and manual testing simply cannotkeepup.Withinisacollectionoftestautomationanecdotes taken from Dorothy Graham’s bookentitled:‘Experiences of Test Automation.’An anecdote is a short account of an incident(especially a biographical one). Numerouspeople told us short stories (anecdotes) of theirexperiences, and because they merit retellingbut don’t constitute full chapters, we collectedthem in this chapter. The stories vary in lengthfrom half a page to five pages. They are allindependent, so they can be read in any order. 29.1 Three Grains of RiceRandy Rice, United StatesConsultant, Speaker, and AuthorAs a consultant, I see a number of differentsituations. The following describes three shortexperiences I have had with a couple of clients.29.1.1 Testware ReviewsI was a consultant once on a project wherewe were trying to bring best practices in testautomation into a large organization thathad only tinkered with test automation. Thecompany’s environment spanned web-based,client/server, and mainframe applications.About 15 test designers and 15 test automatorswere brought in to work on this effort. Thetest tools in use when we first arrived wereold versions not even being supported bythe vendor because of their age. Only a smallportion of the applications were automated toany degree. The automation that was in placeconsisted of large test scripts that were verydifficult to maintain.The project was initiated as one of severalaggressive projects to technically reengineerthe entire IT operation. The chief informationofficer (CIO) who was the original champion oftheseprojectswasabelieverintestautomation.Her successor inherited the projects but did notshare the same commitment and enthusiasmfor many of them. There was also a 6-monthvacancy while the new CIO was being recruited,so things had just coasted along.When the newsheriff came to town, people started trying tofigure out who would survive.Supervising this effort were three senior testautomation consultants who really knew theirstuff and had a very specific approach to befollowed.Wehadsixtestautomationgurusonthemanaging consultant side, and we had regularcommunication based on metrics and goals. Infact, we developed a very nice dashboard thatintegrated directly with the tools. At any timeon any project, people could see the progressbeing made. We gave demonstrations of howthe automation was being created (this wentover management’s heads) and also the resultsof automation, so we had plenty of knowledgeand communication.To their credit, the contracting company trainedall the incoming test design and automationconsultants out of their own pocket. Althoughthese were experienced consultants, thecontractor wanted to set a level baseline ofknowledge for how the work would be done onthis project.After about 3 weeks, it became apparent thatsome of the test automators were going theirown way and deviating from the defined
  3. 3. Test Automation Anecdotes2PAGEapproach. This was a big problem because akeyword approach was being used, and certainkeywords had to work consistently amongapplications. There were too many people whowanted to do things their way instead of theway that had been designed.To correct the issue, the senior consultantsrequired all test designers and consultantsto attend daily technical reviews of testware.Technical reviews are not just for applicationsoftwarecodeorrequirements.Toget30people(more or less) from diverse backgrounds on thesame approach is not a trivial achievement!Before long, this became a peer review type ofeffort, with critiques coming from peers insteadof the senior consultants. It had turned into aforum for collaboration and learning.Good PointReviews of automation testware are beneficialnot just from a technical standpoint but alsofrom an idea-sharing and brainstormingstandpoint.Some of the test consultants resisted thetechnical reviews and didn’t last on the project.They were the same test automators whorefused to follow the designed approach.After a few weeks, it was no longer necessaryto maintain the frequent reviews, and the testautomation effort went a lot more smoothly.Unfortunately, test management and seniortechnical management (at the CIO level) inthis organization never saw the value of testautomation. Therefore, much of the fine workdone by this team was scrapped when seniormanagement pulled all support for this effort.They terminated the contracts of everyonewho knew anything about the automationand ended up “achieving” a negative returnon investment (ROI)—millions of dollars werespent with very little to show for it. I see littlefuture for automation at this company now, inspite of the great work that was done.LessonTechnical success is no guarantee of lastingautomation; management support throughgood communication is needed.This was a huge and very visible project. But thetest manager was like many test managers andhad been thrust into the role with no trainingin testing. The client staff were thin in numbers,skills, and motivation.My bottom line assessment is that theorganization simply was not ready for such anaggressive project. Then, when the sponsoringCIO left, there was no one to champion theproject. Also, the software wasn’t engineeredin a way that was easily automated; it was oldand very fragile. The expectations for ROI werevery high and it would have been better to takesmaller steps first.29.1.2 Missing MaintenanceThere was a move in the late 1990s to go fromfractional stock prices to decimal prices. Fordecades, stock prices had been shown as “$101/2”insteadof“$10.50.”Thereweremanybenefitsto the decimal representation, such as ease ofcomputation, standardization worldwide, andso forth. This was a major conversion effort thatwas almost as significant for the company asthe Y2K maintenance effort.Because the conversion effort was so massiveand time was so short, management decidednot to update the test automation duringthe project. This decision later proved to besignificant.By the time the decimalization project wascomplete, work was well underway for the Y2Kconversion effort.We wanted to update the testautomation for both efforts—decimalizationand Y2K—at the same time. However, theschedule won again, and by the time the Y2Keffort was complete, the test automation was
  4. 4. Test Automation Anecdotes3PAGEdeemed to be so out of date, it would be easierto start all over in a new, more modern tool.This was indeed the case. One of the problemswas the platform, the DEC VAX. There was onlyone tool on the market for that platform. Anemulator-based PC tool could have been used,but then there would be issues of character-based testing.Atthetime,keyword-drivenorevendata-drivenapproaches were not widely known, and theautomators and test managers encountered forthemselves the difficulties of maintaining theautomation code with hardcoded values. Thefirst decision not to keep up with maintenanceof the automated testware proved to be thedeath of the entire test automation effort forthat application. This was a highly complexfinancial application, taking about 3 years tocreate the original test automation. There werenew projects being developed on client/serverplatforms. Starting again from square onemight have been a good idea, but the companyhadn’t yet realized the ROI from the first effort.Basically, the manual test approach just seemedtoo compelling.LessonOnce you abandon the maintenance of theautomation, it is likely to die. For a better chanceof success, choose an automation approachthat will require the least maintenance.29.1.3 A Wildly Successful Proof-of-ConceptI was hired by a large Wall Street companyto assess the quality of its software testingprocess and make recommendations regardinga workable test automation approach. Thiscompany was not new to the idea of testautomation. In fact, it already had three majortest automation tools in place and was lookingfor another test automation solution. Therewas no integration between the various testautomation tools, and they were applied infunctional silos.One particular system at this company wasbeing manually regression tested every day!This one very unfortunate lady performed thesame tests every day for about 8 hours a day.As we were considering which tools might bethe best fit for this system, I suggested that wecontact the various candidate vendors and seeifanywerewillingtosendatechnicalconsultantand perform a proof-of-concept using the thevendor’s tool and my client’s system.My client thought this was an excellent idea, sowe contacted the vendors and found one thatwas willing to send in a consultant at a reduceddaily rate. We felt it was worth the risk to pay forthe proof-of-concept. It would have taken usweeks to try to get an unfamiliar tool working,and we didn’t want to pay for a tool withoutknowing it would work.It seemed to me a good test project for theproof-of-concept was the 8-hour daily manualregression test, so we asked the vendor’stest automation consultant to tackle thatapplication.After3days,theregressiontestswerecompletelyautomated! We were hoping just to get an ideathat the tool would work in the environment.What we got instead was our first success! Weprobably broke even on ROI after 1 month.TipTool vendors can be a great help to get youstarted“on the right foot.”My client was thrilled, I looked good forsuggesting the idea, and the vendor made abig sale. However, the person happiest withthe outcome was the lady who had previouslyperformed the manual regression tests for 8hours a day. Now, she started an automated
  5. 5. Test Automation Anecdotes4PAGEtest and 15 minutes later, the test was done. Hertime was now free for designing better testsand performing more creative tests. 29.2 Understanding Has to GrowMolly Mahai, United StatesTest managerWhen we started looking at automation, Iread everything I could get my hands on. Iknew I couldn’t automate everything. I knewautomation would not replace people, andI knew it would require investment. I readSoftwareTestAutomation(FewsterandGraham,Addison-Wesley, 1999) at a recommendation,and I learned about the steps to take.The funny thing is that even knowing all that,I felt I could not accomplish the first steps toset up an architectural framework. I know thatsounds simple, but what does an architecturelook like? How do we want to capture our tests?How will we set it up so that we can reusescripts? All these questions kept preventing usfrom making progress. So, in my usual fashion,I made a decision and picked an architecture.I had no idea if it would work or not, but weneeded to get moving with something. Thisfreed us up to learn, and learn we did. Wecreated a regression suite that addressed ahandful of our applications, and it looked likewe were moving in the right direction, but weran into problems. There were too many scripts,and the initial grouping (directory structure)was not sufficient for our use.This time, though, I knew a lot more and figuredout that our architecture was lacking. We hadtoomanyprojects,thelibrarywascumbersome,and so on. So, I redesigned the architectureand created staging areas, including a sandboxarea for development scripts, a place for scriptsin use, and a place for scripts that were partof the production suite. We also enforced thenaming conventions that we had put in place.These simple changes fixed a good many of ourorganizational problems.The key is that we knew about this potentialpitfall, and we knew how important it was tohave an architecture that worked for us, but wecouldn’t design that architecture until we knewmore of what we wanted to accomplish. For us,this was not a pitfall that we could avoid: Wehad to learn our way through it. The great thingwas that I was intently aware of this potentialproblem (from reading the book), and I keptmy eye on it. We redesigned the architectureas soon as we realized it wasn’t working for us,and the impact on our automation effort wasnegligible.I relate this to trying to explain to teenagersthat they will view things differently whenthey are older. They cannot grasp it throughhearing from someone else; they must learn itfor themselves.LessonExperiencing the problem is often the best (oronly) way to see a better solution. 29.3 First Day Automated TestingJonathon Lee Wright, United KingdomConsultant and speakerIn the past decade, I have dealt with a numberof ways of creating testware frameworks andfound advantages and disadvantages in eachof the following approaches:• Modularity driven• Keyword driven
  6. 6. Test Automation Anecdotes5PAGE• Data drivenIn2009,Iwastaskedwithautomatingthetestingof a new package for the New Zealand LotteriesCommission. The requirements phase had onlyjust been completed, and the scheduled testingwas looking very tight—less than a month (theschedule was imposed on us by marketing, whohad already overcommitted their advertising!).With just 2 months until the release would bedelivered, this was an opportunity to adaptthe testware frameworks I had used before,combining them with the latest technologies todevelop what I refer to as a hybrid (keyword-/data-driven) automation framework.Not only would this meet all the requirementsset by the business, but more important, itwould allow us to start the development of thetestware framework immediately.29.3.1 Initial InvestmentThe inherent problem with the hybrid approachisthelargeinitialinvestmentintimeandefforttodesign the testware framework. Consequently,it was important that the development of thisframework be treated as a legitimate projectin its own right with its own planning, design,development, and testing phases together withthe necessary resources. Table 29.1 shows ourestimates for the effort required to create 100test scenarios.In essence, the hybrid approach wouldtake roughly twice as long as the previousautomation approach, which in turn would taketwice as long to automate than the preparationfor manual testing.29.3.2 What Is to Be Automated?Given the limited time available and theincreased initial effort required, it was criticalthat we identified the optimum test coverage.To avoid developing unnecessary testcomponents, we used the MoSCoW method:• What must be automated?• What should be automated?• What could be automated?• What won’t be automated?This method allowed us to focus our efforts onthose test components that were necessaryand would give the greatest value by assessingthe associated business risk, reusability,usage, complexity, and feasibility of each testcomponent.Thetestcomponentswereregardedasindividualjigsaw pieces, but we kept in mind what wasneeded to complete the entire puzzle.Business process modeling (BPM) was usedwithin the centralized test management tool(Quality Centre) to represent the individualpuzzle pieces (test components); mappingthe pieces revealed the workflow through thesystem.Figure 29.1 shows how one BPM may onlyinclude 5 puzzle pieces but enable more than20 different paths through the system undertest (SUT), with each path having a differentbusiness risk and impact assessment.This made it easier to decide which automationmodules to develop first by starting small andautomating only the most critical businesscreate 100 test scenarios.In essence, the hybrid approach would take roughly twicautomation approach, which in turn would take twice as lonpreparation for manual testing.29.3.2 What Is to Be Automated?Given the limited time available and the increased initial effocal that we identified the optimum test coverage. To avoidtest components, we used the MoSCoW method:What■ must be automated?What■ should be automated?What■ could be automated?What■ won’t be automated?Table 29.1 Simple Calculations for Initial Testware Preparation EApproach EffortManual 2 weeksExisting framework 1 monthHybrid framework 2 monthsTable 29.1 Simple Calculations for Initial TestwarePreparation Effort
  7. 7. Test Automation Anecdotes6PAGEprocesses—keeping it as simple as possiblewhile recognizing that the test coverage couldbe increased as the framework matured overtime.The decomposition of the workflows into ahigh-level model visualizes and enables an agileapproach to the framework development. Thedevelopment’s build order and resource focusbecomes obvious.Each path through the system represents anagile feature, which may be in or out of scopedepending on time and resources. Anotherbenefit of this approach is that the modelbecomesanartifactthatmaybesharedbetweenthe test framework and target applicationdevelopers.Good PointStart by automating the most valuable tests,but plan for later growth.29.3.3 First Day Automated TestingThe key to first day automated testing is tocreate a dynamic object repository based ona combination of fuzzy logic and descriptiveprogramming supporting the design anddevelopment of test scenarios before the actualdelivery of the SUT.Traditionally, because of the dependency onbuilding the object repository, test automationis carried out at the end of the softwaredevelopmentlifecycleoncetheSUTisdelivered.However, because we had only a single monthin which to execute testing but a full 2 monthsbefore the SUT was delivered, it seemed logicalto develop the testware framework beforehandwhile the application was still in development.Good PointAutomation can (and should) start beforethe software being tested is delivered, so theautomated tests are ready to run on the first daythe software is released. But this requires goodplanning and good testware architecture. Business-Level KeywordsTo allow the creation of test scenarios ahead oftheSUTdelivery,ahigh-levelkeywordapproachwas used to represent:• Specific BPM and business process testing (BPT) modulesnents); mapping the pieces revealed the workflow through the system.Figure 29.1 shows how one BPM may only include 5 puzzle pieces but enablemore than 20 different paths through the system under test (SUT), with each pathhaving a different business risk and impact assessment.This made it easier to decide which automation modules to develop first by start-ing small and automating only the most critical business processes—keeping it assimple as possible while recognizing that the test coverage could be increased as theframework matured over time.The decomposition of the workflows into a high-level model visualizes andenables an agile approach to the framework development. The development’s buildorder and resource focus becomes obvious.Each path through the system represents an agile feature, which may be in orout of scope depending on time and resources. Another benefit of this approach isthat the model becomes an artifact that may be shared between the test frameworkand target application developers.A1B1Login.ProcessBuy.TicketsCheck.TicketsMy.AccountLogout.ProcessD1C1B2B4C3 B3C2B5D3C4 D2D4 D5C5figurE 29.1 Business process model of the SUT.Figure 29.1 Business process model of the SUT.
  8. 8. Test Automation Anecdotes7PAGE• Specific/collections of user stories• Assigned work items• Queries against the test asset databaseUsing high-level business-level keywords, suchas Login.Process, allows complexity hiding andreusable encapsulation. Login.Process containsa number of low-level keywords, such as EnterUsername Text and Press Login Button.The collection of application keywordsrepresents natural domain-specific languagesthat translate into a number of lower-levelactions performed before and after the coreevent. This included checking that the pretestand posttest conditions were met and theactions and reactions, including any popup/error recovery, were processed correctly.Good PointThe more automation code is reused, themore worthwhile it is to build in recovery fromunexpected errors or events and the morerobust the scripts are.Usingthisapproachmeantwehadafoundationupon which to design and develop reliable,domain-specific, and reusable test scenariosbefore the release of the SUT.Writingthetestscenarios(manualorautomated)as business-level keywords combined withnatural language made it accessible toapplication domain experts, business analysts,and developers. The test language was self-validatingandhumanreadable,whichremovedthe requirements to educate the end user in thetool.Theverbsandnounsinthedomain-specificlanguagewerewritteninnaturallanguageusingcontext-sensitive validation. This improved theutilization of resources by encouraging jointcollaboration between multifunctional teamswhilesupportingbehavior-drivendevelopment(BDD).The Scrum team was made up of a numberof multidiscipline team members (businessanalysts, testers, and developers) sharing thevarious roles of the test design phase withoutany previous automation experience. Thisallowed teams to collaborate and share testassets such as BPM/BPT modules, user stories,and work items. They could also run queriesagainst previously created test scenarios andreuse shared test cases and steps.The flexibility of having the test scenariosstored in a database also allowed for full/partialfallback support for legacy manual testingbecause the test data could be easily extractedintoatraditionaltestscenarioformat.Itwaseasyto read because of the use of natural languagecombined with valid test data that could beeasily used in manual test execution.Good PointAutomated tests should be accessible andreadable by all and should enable the tests tobe run manually if necessary.In summary, this approach of managingcentralized test assets to generate sanitizedtest scenarios validated against business rulesprovided ready-to-use tests and data in thecorrect format. This was evidenced by theability to generate tests featuring over 10,000ticket number combination states coveringall possible combinations of ticket types andamounts (this excluded specialized test runssuch as boundary and negative tests, whichwere run separately) before the SUT had evenbeen written.29.3.4 Problems and SolutionsWe found problems stemming from proceduresnot being followed consistently. For example,changes to the functionality of reusable testcomponents’ jigsaw pieces were not beingchecked in correctly. This was caused by
  9. 9. Test Automation Anecdotes8PAGEnot having an enforceable gated check-inprocedure and consequently resulted in limitedreusability of some of the test components. Theproblem was solved by enforcing the check-inprocedures in the Configuration Managementtool.LessonAutomation development requires the samediscipline as software developmentIt became apparent when the testwareframework entered the execution phase andwas distributed across a pool of remote testclients generated by a virtual machine (VM)dispenser that there was limited direct visibilityinto the test execution status.While it was relatively easy to identify primarycore modules failing on startup, more subtlechanges to reusable test components weremuch harder to spot. The requirement for a testasset loader to validate the current SUT buildagainstthetestassetdatabasebeforeexecutioncould have prevented this.Without the ability to monitor runtime failure,especially controlled failure (i.e., scenariorecovery), a significant amount of executiontime was wasted. For example, a discretechange to a test component could cause a false-positive error, which in turn caused the testwareframework to repeatedly retry the current testscenariobeforeattemptingtocontinuethroughthe remaining test cases.What was needed herewas a suitable real-time dashboard that couldprovide notifications regarding the SUT healthas well as progress of test client execution.We solved this problem by devising a way toflag the overall status of a test set— In Progress,Ready, Repair, or Blocked—to keep the testerinformed. This would affect the current test runand associated test client VM’s state where, forexample, a Blocked status did not allow the testrun to be continued until the necessary pretestconditions were met (e.g., the Lotto productdraw had been successfully processed).TipKeeptrackofthingsthataffectthetestexecutionto avoid wasting time running tests that will failfor reasons you are not interested in. There is adanger, however, that tests that are turned offwill never be turned back on.29.3.5 Results of Our First DayAutomation ApproachThis approach worked extremely well, and werealized a good return on investment (ROI)for the additional effort in developing theframework.Once the release was delivered, the executionwas run constantly, day and night. This wasmade possible by having dedicated resourcesavailable during the day to deal with basicdebugging of failed scripts and execution.Developers based in another time zone werealso available in the evening to maintain theframework and provide additional support forimproved test coverage.Overall, this approach was found to work wellin this case study by demonstrating its innateadvantages, reflected in what I like to callthe approach: Hybrid Keyword Data-DrivenAutomation Framework.• Hybrid: Utilizing the best technologies and resources to do the job.• Keyword: Creating simple and robust test scenarios written in business-level keywords combined with natural language.• Data: Effective use of dynamic business data to provide an input source.• Driven: Reusable component modules and libraries to provide reliable processing of generic actions, objects, and events.• Automation: That is collaborative,
  10. 10. Test Automation Anecdotes9PAGE distributed, and scalable.• Framework: Independent of application, technology, or environment under test.The best aspects of these proven approachesdemonstrate how they have evolved overthe past decade; this echoes some of theprogress toward leaner and more agilebusiness methodologies. They are in a constantstate of evolution—just as the underpinningtechnologies evolve over time.A significant benefit was that the frameworkhad the ability to support multiple browsers,platforms, and technology stacks under aunified engine with the capability to deal withgeneric object classes as well as application-specific classes. 29.4 Attempting to Get Automation StartedTessa Benzie, New ZealandTest engineerMy company was keen to get into testautomation. We had a couple of days ofconsultancy (with one of the authors of thisbook) to explore what we wanted and thebest ways of achieving our goal. We discussedgood objectives for automation and createdan outline plan. We realized the need for achampionwhowouldbetheinternalenthusiastfor the automation. The champion needed tobe someone with development skills.We hired someone who was thought to be asuitable candidate to be the champion for ourinitiative. However, before he could get startedon the automation, he needed to do sometraining in databases, as this was also a requiredarea of expertise, and we needed him to be upto speed with the latest developments there.After the training, he was asked to helpsome people sort out some problems withtheir database, again “before you start onthe automation.” After these people, therewere some other groups needing help withdatabases. When do you think he finally startedautomation? As you may have guessed already,he never did! Of course, it can be very hard fora new person to say no, but the consequencesshould be pointed out.A few months after that, a new test managercame in who was keen to get the automationstarted at the company. He did some greatwork in pushing for automation, and we choseand acquired a test tool that looked suitable.There were a couple of other contractors (thetest manager was also a contractor) who werecoming to grips with the initial use of the tooland began to see how it could be beneficial.So we had a good start, we thought, to ourautomation initiative.Shortly after this, there was a reorganization,and the contractors and test manager werelet go. A new quality assurance manager wasbrought in, but test automation was not onher list of priorities. Some people were tryingto use some of the automated tests, but therewas no support for this activity. However, therewere many, many tests that needed to be doneurgently, including lots of regression tests.Now we had“football teams”of manual testers,including many contractors.LessonOrganizational support is needed to get asignificant automation initiative started; be sureyou choose a champion who will stay on taskand be the driving force behind your initiative.Beware of knowledge walking out the doorwith your contractors— it’s far better to involvepermanent staff.
  11. 11. Test Automation Anecdotes10PAGE 29.5 Struggling with (against) ManagementKai Sann, AustriaEngineer and test managerI have had some “interesting” managers overthe years who had some challenging effects onthe way we did test automation.29.5.1 The“It Must Be Good, I’ve AlreadyAdvertised It”ManagerWestartedsoftwaretestautomationin2002.Themanagement’s intention was to reduce time forthe system test. Furthermore, the managementused automation as a marketing strategy beforethe automation was developed.At this time, there was no calculation for returnon investment (ROI). The approach was this:Software test automation must pay becausemanual testing is no longer needed. The goalwas to automate 100 percent of all test cases.I had a hard time explaining that software testautomation is just one of many methods toachieve better software and that it is not free—or even cheap.29.5.2 The “Testers Aren’t Programmers”ManagerWe started very classically and believed thepromises of the vendors.They told us,“You onlyneed to capture and replay,” but we found thiswas not true. In our experience, this leads toshelfware, not success—it does not pay off.After some trial and mostly error, we startedto write automation code. At this point, wewere far away from developing automated testcases. We needed some lessons in writing code.We were lucky to have very good mentors onour development team who taught us to writelibraries and functions so we didn’t have to codethe same tasks repeatedly.I had a discussion with my boss about whatprogramming is. He explained to me thathe consciously hadn’t hired testers with aninformatics degree because he didn’t want tohave more developers in his department. Youcan imagine his surprise when I told him thatour automation code included libraries andfunctions.He told his superiors that the testers do“advancedscripting”ratherthancodingbecausehe was afraid that the testers would otherwisebe forced to write production code!Good PointBeware of political issues and fears.29.5.3 The“Automate Bugs”ManagerAn idea provided by one manager was toautomate bugs we received from our customercare center. We suffer the consequences to thisday. How did this work? Our developers had tofix our customers’ bugs. We were told to readthis bug and automate this exact user action.This is where the consequences come in: Notknowing any better, we hardcoded the userdata into our automation. After 2 years, we wereone major release behind the development. Wedidn’t know anything about datadriven tests atthat time.We were automating bugs for versions thatwere not in the field anymore. Most of these testcases still exist because we haven’t had time toreplace them.
  12. 12. Test Automation Anecdotes11PAGELessonHolding the title of manager doesn’t necessarilymean a person knows best (or anything) abouttest automation!You may need to educate yourmanager.29.5.4 The “Impress the Customers (theWrong Way)”ManagerMy boss had the habit of installing the untestedbeta versions for presentations of the softwarein front of customers. He would install unstableversions and then call our developers from theairport at 5:30 a.m. and order immediate fixesto be sent to him by email.Our programmers hated this so much that weintroduced an automated smoke test. This testchecks if we have a new build; then it installs thebeta, and finally it checks the basic functions ofour product. Our boss was told to only installsmoketested beta versions.Today we don’t have this boss issue anymore,but we continue the automated smoke tests forour nightly builds because they provide us witha good guess about the state of our software.Here we really save money because smoke testsmust be done anyway and we can provide ourdevelopment with a lot of issues concerningthe integration of new modules at an earlystage. We expand this test every few months.The coolest thing is that we are informed aboutthe latest test results by email.So in spite of some challenging managers, weare now doing well with our automation!Good PointSometimesanapproachadoptedforonereasonturns out to be good for other reasons. 29.6 Exploratory Test Automation: Database Record LockingDouglas Hoffman, United StatesConsultant and speakerPre–World Wide Web (late 1980s), I was thequality assurance (QA) manager, test lead, andtest architect for the second-largest relationaldatabase management system (RDBMS)software company at the time. Before landingthe job, I got my bachelor’s degree in computerscience, started out as a systems engineer,and had worked my way up into hands-onengineering services management (QA, techsupport, sales support, and tech writing).The company had about 300 employees,mostly in San Mateo, California. The relationaldatabase engine had evolved over more than10 years and had a wellestablished, fairly stablecodebase.Atthetime,thecodehadtobeportedacross 180 hardware/software platforms (mostwere variations of UNIX). The QA team wassmall (initially with a ratio of about 1:20 withdevelopers, growing to ~1:5 over 18 months)and nearly completely focused on testing.Most new testers were recruited from othercompanies’development organizations.To support the large number of versions, theproduct was developed using internal switchesto enable or disable different code functions.Therefore, most of the porting was done byselecting the proper combinations of switches.This meant that once features or bug fixes hadbeen incorporated into the base code, thevarious ports would pour into QA.Because the test team was small, we spentalmost all our time running tests on the various
  13. 13. Test Automation Anecdotes12PAGEplatforms. Little time was available for designand implementation of tests for new features.A thorough test run for a new release on aplatform could take 2 weeks, and the dozentesterscouldreceive150versionsinafewweeks.The management and technical challengesdealing with this situation are other casestudies in themselves. This case study focuseson one particular exploratory automated testwe created.Unlike regression tests that do the same thingevery time they are run, exploratory automatedtests are automated tests that don’t havepredetermined results built in; they createnew conditions and inputs each time they’rerun. Exploratory test automation is capableof finding bugs that were never specificallyconceived of in advance. These are typicallylong-sequence tests and involve huge numbersof iterations because they are only limited byavailable computer resources.TipWhen the input space of an application is huge,exploratory automated tests can find defectsthat are difficult to find manually.29.6.1 The Case StudyBug being sought: Errors in database recordlocking (failure to lock or release a lock on arecord, table, or database).When I took over the QA organization, theexisting test cases were simple applicationswritten in the various frontend languages,mostly our proprietary SQL. Most existing testswere applications and data sets collected fromcustomers or scripts that verified bug fixes.Automation was achieved using a simple shellscript that walked through the directoriescontaining the tests and sequentially launchedthem. Most of the testers’ efforts were inanalyzing the results and tweaking the teststo conform with nuances on the variousplatforms.Oneareaofconcernthatwasn’twelltestedusingour regression tests was record locking. A fewintermittent bugs that locked up the programsor clobbered data had been found in the field.The locking mechanism was complex becauseof the distributed nature of the database. Forexample:• Parts of the database might be replicated and the“master” copy moved around as requests were made.• Different parts of the database needed to communicate about actions before they could be completed.• Requests could occur simultaneously.• Common data could be updated by multiple users.• One user’s request for exclusive access (e.g., to update part of a record) could cause some other users’requests to wait.• User requests for nonexclusive access might proceed under some circumstances.• Non-overlapping parts of interlocking records might be updated by different users.• Timeouts could occur, causing locks and/or requests to be cancelled.Most multiuser database systems at the timewere hardwired and LAN based, so Internet andcloud issues weren’t part of the picture. (Thiswas before widespread Internet use, browsers,or the World Wide Web.) Frontend programswere built out of compiled or interpretedprograms written in proprietary languages. Theuse of LANs meant that interrupt events camedirectly through hardware drivers and were notprocessed by higher-level system services.Prior to the test automation project, thestraightforward locking sequences were testedusing manual scripts on two terminals. Forexample,
  14. 14. Test Automation Anecdotes13PAGE1. One user would open a record for nonexclusive update (which should lock the record from being modified but allow reading of the record by other users).2. A second user would open and attempt to modify the record (thus successfully opening but then encountering a record lock).3. Another test would have the first user opening for nonexclusive read (which should not lock the record and should allow reading and modifying of the record by other users).4. The second user would read and update the record (which should work).The regression tests confirmed the basicfunctioning of the lock/unlock mechanisms.Only a subset of condition pairs could bemanually tested this way because of theamount of time it took, and complex sequencesof interactions were out of the question. Figure29.2 shows an example of the interaction of twousers attempting to access the same record.In relational databases, updating a record canwrite data in many tables and cause updatesto multiple indexes. Different users runningdifferent programs may reference somecommon data fields along with unique data.The data records are contained in multiple files(called tables), and programs reference somesubset of the fields of data across the database.Intermittent, seemingly unreproducibleproblems could occur when the requestsoverlapped at precisely the wrong times. Forexample, the second read request might comein while the first was in the process of updatingthe database record. There might be tinywindows of time during which a lock might bemissed or a partially updated record returned.These kinds of combinations are extremelydifficult to encounter and nearly impossible toreproduce manually. We decided that the bestway to look for errors was to generate lots ofdatabase activitiesfrom multiple users at thesame time.TipWhen the input space of an application is huge,exploratory automated tests can find defectsthat are difficult to find manually.The challenge was to create multithreaded teststhat could find timing-related problems of thistype. The goal was to produce tests that wouldgeneratelots of conditionsand could detect thebugs and provide enough failure information tothe developers so they could isolate the causeand have some chance at fixing the bugs.Automated tests: We created a single programthataccessedadatabaseusingvariousrandomlyselected access methods and locking options.The test verified and logged each action. Wethen ran multiple instances of the programat the same time (each being its own threador on its own machine). This generated hugenumbers of combinations and sequences ofdatabase activities. The logs provided enoughinformationtorecreatethe(apparent)sequenceof database actions from all the threads. If noproblems were detected, we discarded thelogs because they could get quite large. Whilethe tests were running, a monitoring programobserved the database and log files to ensurethat none of the threads reported an error orbecame hung.29.6 Exploratory Test Automation: Database Record Lockingthe second read request might come in while the first was in the process of updathe database record. There might be tiny windows of time during which a lock mbe missed or a partially updated record returned. These kinds of combinationextremely difficult to encounter and nearly impossible to reproduce manuallydecided that the best way to look for errors was to generate lots of database activfrom multiple users at the same time.Good PointGood candidates for automation are tests that are difficult to run manually andthose that are too complex for manual testing.The challenge was to create multithreaded tests that could find timing-relproblems of this type. The goal was to produce tests that would generate loconditions and could detect the bugs and provide enough failure information todevelopers so they could isolate the cause and have some chance at fixing the buAutomated tests: We created a single program that accessed a database using varrandomly selected access methods and locking options. The test verified and loeach action. We then ran multiple instances of the program at the same time (User A User BDatabaseLock RecordLock RecordLockedBlockedUnlock RecordLockedUnlocked{Pause}Unlock Record{Continue}Record Locking ExampleUnlockedfigurE 29.2 Record locking exampleFigure 29.2 Record locking example
  15. 15. Test Automation Anecdotes14PAGEOracle:Watchingforerrorreturnsandobservingwhether any of the threads terminated or tookexcessively long. The test program did verytrivial verification of its own activities. By thenature of multiple users updating overlappingdata all the time, data changes might be madeby any user at any time. We couldn’t reliablyconfirm what was expected because someother user activity might change some data bythe time the test program reread it. Becausemost database activities completed in fractionsof seconds, if there was no lockout, the monitorprogram checked for multisecond delays onnonlocked transactions or after locks had beenlogged as released.Good PointA test oracle, especially for automatedexploratory testing, may just be lookingfor system failures rather than functionalcorrectness. Use an oracle that is appropriateto the testing you are doing.Method summary:1. Created a program that independently, randomly accessed records for update, read, insert, and delete. Different types of record locking were randomly included with the requests. Each program logged its activity for diagnostic purposes and checked for reasonable responses to its requests.2. Ran multiple instances of the program at the same time so they potentially accessed the same data and could interfere with one another. Access should have been denied if records were locked by other processes and allowed if none of the other threads was locking the referenced record.3. Created and ran another program to monitor the log file and the database engine to detect problem indications and shut down if a problem occurred.Because the threads were running doingrandom activities, different combinations andsequences of activities were generated at a highrateofspeed.Theprogramsmighthavedetectederrors, or the threads might hang or abnormallyterminate,indicatingthepresenceofbugs.Eachinstance of the test generated large numbersof combinations of events and different timingsequences. The number of actions was limitedby the amount of time we allocated for the locktesting. We sometimes ran the test for a fewminutes, but at other times it could run an houror longer. Each thread might only do hundredsof database actions per second because of thetime it took for waiting, error checking, andlogging. We ran from three to a dozen threadsusing multiple networked systems, so a minuteof testing might generate 100,000 to 300,000possible locking conditions.Results: We caught and were able to fix anumber of interesting combinations, timingrelated bugs, and one design problem. Forexample, a bug might come up when:• User A opens a record for update.• User B waits for the record to be unlocked to do its own update.• User C waits for the record to be unlocked to do its own update.• User A modifies the data but releases the lock without committing the change.• User B modifies the data but releases the lock without committing the change.• User C updates the data, commits it, and releases the lock.• User C’s data was not written into the database.I was surprised because a few of the timing- andsequence-related bugs were not related to therecord locking itself. If the commits occurredat the same time (within a tiny time window),the database could become corrupted bysimultaneous writes of different records bymultiple users in a single table. Although therecords were not locked because the userswere referencing different rows, the data couldbecome switched.
  16. 16. Test Automation Anecdotes15PAGEWe couldn’t be certain that we caught all thebugs because of the nature of these kindsof timing- and sequence-related problems.Although millions of combinations were triedand checked, there were myriad possibilities forerrorsthatwecouldn’tdetectordidn’tcheckfor.Trillions of combinations wouldn’t amount to ameasurable percentage of the total number ofpossibilities. But, to the best of my knowledge,there were no reported field problems of thatnature for at least a year after we created thesetests. (I moved on and no longer had access tosuch information.)Good PointThe benefits of exploratory automated testsmay be significant even if you can’t know whatyou didn’t test.We didn’t leave reproducibility to chance,although even running the same series ofinputs doesn’t always reproduce a problem.Theat-random tests used pseudorandom numbers;that is, we recorded seed values to be ableto restart the same random sequences. Thisapproachsubstantiallyimprovesreproducibility.We generated a new seed first when we wantedto do a new random walk, and we reused theseed to rerun a test.TipEven random tests should be reproducible inorder to confirm bugs and bug fixes. Use thesame starting point (the “seed”) to reproducethe same random test.The archiving system is critical for tracing backto find the likely cause and also as the testoracle. Much of the data being recorded can bechecked for consistency and obvious outliersto detect when likely bugs are encountered. Itend to log the things developers tell me mightbe important to them as the test progressesand then “dump the world” when a bug issuspected.We used some functional test suites that wereavailable for SQL, including from the NationalBureau of Standards (which later became theNationalInstituteofStandardsandTechnology),but they only checked basic functionality. Weused them to some degree, but they were notrandom,asynchronous,ormultithreaded.TPC-Bwasn’t created until several years later.Recognizing the root cause of reported bugsrequired serious investigation because thefailures we were seeking generally requiredmultiple simultaneous (or a specific sequenceof) events. Many of the factors we looked atwere environmental. We were primarily lookingfor fundamental violations of the locking rules(deadlocks and data corruption), so recognizingthose failures was straightforward. Identifyingthe cause was more difficult and frustrating. Itsometimes took a lot of investigation, and oncein a while, we gave up looking for the cause.Thiswas frustrating because we knew there was abug, but we couldn’t do anything about it otherthan look for other symptoms. Most of the time,though, the developers were able to identifythe probable cause by looking for the possibleways the failure could have happened.Good PointUnreproducible failures are worth reportingbecause sometimes developers can trace thecause in the code if they know that somethingis wrong.
  17. 17. Test Automation Anecdotes16PAGE 29.7 Lessons Learned from Test Automation in an Embedded Hardware–Software Computer EnvironmentJon Hagar, United StatesEngineer, trainer, and consultantEmbedded systems comprise specializedhardware, software, and operations. They comewith all of the problems of normal software, butthey also include some unique aspects:• Specialized hardware that the software “controls”with long and concurrent development cycles.• Hardware problems that are“fixed”in the software late in the project.• Limited user interfaces and minimal human intervention.• Small amounts of dense, complex functions often in the control theory domain (e.g., calculating trajectories, flight dynamics, vehicle body characteristics, and orbital targets).• (A big one) Very tight real-time performance issues (often in millisecond or microsecond ranges).Products that make up embedded softwaresystems now span the automotive, control,avionics, medical, telecom, electronics, andalmost every other product domain one canthink of. I have been involved in space avionics(guidance, navigation, and control software),butmanyoftheapproachesandlessonslearnedare applicable to other embedded softwaresystems. In this section, we use examples drawnfrom a hypothetical but historically based spaceflight software embedded system.The goal of verification, validation, and testing(VV&T) is to show that embedded softwareis ready for use and the risk of failure due tosoftware can be considered acceptable by thestakeholders.Development programs can be small—forexample, 30,000 source lines of code (withstaffs of 10 to 60 people)—yet these programsare time and computationally complex and arecritical to the successful control of the hardwaresystem.29.7.1 VV&T Process and ToolsWe typically have four levels of testing andtools that support each level. The lowest levelis probably the most different for embeddedsystems because it is nearest to the hardware.It uses a host/target configuration and cross-compiled code (including automation code).Cross-compiling is where source code iscompiled on one computer, not into thebinary (executable) of that (host) computerbut rather into binary executable code thatwill run on a different computer (the “target”)that is too limited to be able to run a compileron. Our testing at this level aims to checkagainst standards and code coverage as well asrequirements and design and is automated bythe developer.We call this “implementation structuralverification testing” (some places call this unittesting). This testing is conducted with a digitalsimulation of the computer and/ or a single-board target hardware-based computer.The implementation test tools were customizedin the beginning, but other off-the-shelf toolswereaddedlater.ExamplesincludeLDRATBrun,Cantata, and AdaTEST. The project used bothtest-driven development and code-then-test
  18. 18. Test Automation Anecdotes17PAGEimplementation approaches. The comparisonand review of results, which include verycomplex calculations, is done using test oracleinformation generated from commercial toolssuch as MATLAB, BridgePoint, and Mathmatica.The middle level, which we call design-basedsimulation tools, uses tools that are based onsoftware architecture structures and designinformation, which have been integrated acrossmodule boundaries. These tools allow theassessment of software for particular aspectsindividually. In some projects, model-baseddevelopment tools, BridgePoint, and MATLABwere used, and this enabled the integrationefforts to go better than in past systems,because the models enforced rules and checksthat prevented many integration defects fromhappening in the first place.TipUsing models can help to prevent and eliminatedefects that otherwise would be found intesting (or not found), but nothing guaranteesthat you find 100 percent of them.The next level is requirements-based simulation(scientific simulation tools). These simulations(driven by models) are done in both a holisticway and based on individual functions. Forexample, a simulation may model the entireboost profile of a system with full vehicledynamics simulation, and another simulationmay model the specifics of how the attitudethrust vector control works.Thisallowssystemevaluationfromamicroscopiclevel to a macroscopic level. The results fromone level can be used as automated oracles toother levels of VV&T test supporting “compare”activities.This approach of using simulation/models todrive and analyze test results comes with arisk. There is the chance that an error can becontained in the model or tool that replicatesand “offsets” an error in the actual product(a self-fulfilling model result). This is a classicproblem with model-based test oracles. Tohelp with this risk, the project used the levels oftesting (multiple compares), a variety of tools,different VV&T techniques, and expert skilledhuman reviewers who were aware of this risk.Thesemethods,whenusedincombinationwithtesting, were found to detect errors if they exist(one major objective) and resulted in softwarethat worked.Finally, at a system level, VV&T of the softwareusesactualhardwareintheloopandoperations.An extensive, real-time, continuous digitalsimulation modeling and feedback systemof computers is used to test the software in arealistic environment with the same interfaces,inputs, and outputs as in the actual system.The system under test runs in actual real time;thus there is no speed-up or slow-down oftime due to the test harness. Additionally, withhardware in the loop and realistic simulations,complete use scenarios involving the hardwareand software could be played out with both fortypical usage scenarios (daily use) and unusualsituations such as high load, boundary cases,and invalid inputs.29.7.2 Lessons LearnedThis section summarizes some generalobservations that the projects had during theinitial setup and use of automated VV&T tools:• Training: It is important to allow both time and money for training on tools and testing.• Planning: Tools must be planned for and developed like any software effort. Automated VV&T tools are not“plug and play.”To be successful, plan for development, establish a schedule and budget, integrate with existing processes, plan the test environment, and also test the test tools. Test tools must be“engineered”l
  19. 19. Test Automation Anecdotes18PAGE like any development effort.• Have an advocate: Test tools need a champion in order for them to become incorporated into common use. The champion is someone who knows the tools and advocates their use. Success comes from getting staff to think“outside the automated tool box.”The new tools must“integrate”with the existing staff, which means education, mentoring, and some customization. Advocates work these issues.• Usability of a tool must be reasonable for the users: While people will need training on tools, and tools by nature have complexities, a tool that is too hard to use or is constantly in revision by vendors leads to frustration by users that, in the extreme, will lead to shelfware. Ensure that the user interface is part of the selection evaluation before purchasing any tool.Good PointUsability of the tools is important—even for“techies.”.• Expect some failures and learn from them: Our project explored several tools that were abandoned after an initial period of time. While failure is not good, it is really only total failure when one does not learn from the mistake. Also, management must avoid blaming engineers for the failure of an idea because doing so stifles future ideas.Good PointIf you learn from your mistakes, you have nottotally failed. Any failure or mistake becomes asource of information to share.• Know your process: Automated test tools must fit within your process. If you lack process, just having tools will probably result in failure. Expect some changes in your process when you get a new tool, but a tool that is outside of your process will likely become shelfware.• Embedded systems have special problems in test automation: Despite progress, automated test tools do not totally solve all embedded VV&T problems. For example, our projects found issues in dealing with cross-compiling, timing, initialization, data value representation, and requirements engineering. These can be overcome, but that means vendors have more functions to add and projects will take more money and time. Plan for the unexpected.• Tools evolve: Plan on test tool integration cycles with increments.• Configuration management (CM): Even with VV&T tools, projects need to manage and control all aspects of the configuration, including the test tools as well as the test data.29.7.3 Summary of ResultsAlthough I am not permitted to reveal specificdata, when compared to custom-developedtools and manual testing, establishingan automated commercial-based VV&Tenvironment took about 50 percent fewerpeople. The projects tend to take these savingsto create more and/or better automated tests.While adding to test automation, the projectsmaintained and improved functionality andquality. Further, maintenance- regression costsdecreased because vendors provided upgradesfor a low annual fee (relative to staff costs forpurely customized tools). Commercial toolshave a disadvantage of lacking total projectprocess customization, but this has proven tobe a minor issue as long as the major aspects ofthe process were supported by the tools.Additionally,theprojectsreducedteststaffworkhours by between 40 and 75 percent (basedon past VV&T cycles). We found that our test
  20. 20. Test Automation Anecdotes19PAGEdesigners were more productive. We createdthe same number of tests and executed themin less time and found more defects earlier andfaster. We had fewer “break-it, fix-it” cycles ofregression testing, which meant that less effortwas needed to achieve the same level of qualityin the testing and the same defect detectionrates.In an embedded software VV&T environment,automatedtesttoolscanbegoodifyouconsiderthem as tools and not “magic bullets.” Peoplemake tools work, and people do the hard partsof VV&T engineering that tools cannot do. Toolscan automate the parts humans do not like orare not good at. Embedded projects continueto evolve VV&T automation. VV&T automationtools take effort, increments, and iterations.Tools aid people—but are not a replacementfor them.Good PointThe best use of tools is to support people. 29.8 The Contagious ClockJon Hagar, United StatesEngineer, trainer, and consultantSometimes a good testing idea is contagious.Once it meets one need in your system, otheruses may emerge that were quite unexpectedwhen you began.29.8.1 The Original ClockI had just been hired at Google as a developeron a project during its preparation for publicrelease. The system under developmentembeddedatimestampwhenrecordingcertainevents. Depending on how long it had beensince the events were recorded, the systemneeded to present, interpret, or process theevents in different ways.The project had a strong mandate fordevelopers to demonstrate the features theycreated via automated unit and system tests.As my first development task, I took on the jobof designing and coding an application clock tomake developer testing of time-based behaviorsimpler. In production, the application clockfollowsthesystemclock,butfortesting,itwrapsa test clock that can be manipulated to simulatethe passing of minutes, hours, or days.29.8.2 Increasing UsefulnessAt first, the application clock was used forautomated testing of portions of codeencapsulating the core logic for time-dependentfeatures.However,thesystemunderdevelopment could be driven as a whole viaa scripting language that could simulate oneor more users interacting with the system toaccomplish a set of tasks. Script-driven systemtests were the common property of developers,featureowners,andateamoftestingspecialists.The testing team used script-driven systemtests alongside manual verification to exercisethe system in detail before each version wasreleased to production. Soon I helped addcommands to the scripting language to controlthe clock, allowing nonprogrammers to set upscenarios that included the passage of time.29.8.3 Compelling PushThe original application clock was limitedby design so that the clock could never bemanipulated in the production system andthereby create troublesome inconsistencies.However, the testing team needed to exercisethe system and its features interactively in a
  21. 21. Test Automation Anecdotes20PAGEstaging environment similar to the productionsetup.However,fortestingtime-basedbehavior,sometimes they set up a scenario before aweekend and returned after the weekend toverify the correct behavior. Other times thetesting team changed the system clock so thatthe application would pick up the changedtime and demonstrate the desired behavior.Both of these techniques were laborious anderror prone, with the system clock manipulationfrequently causing side effects that would ruinthe test.At the request of a primary tester and anotherdeveloper familiar with the application clock,I revisited the application clock design. By thistime, the system supported a mechanism forenabling and disabling features in productionwithout having to redeploy a new systembinary. This mechanism allowed me to guardthe application clock from being manipulatedon the actual production servers while allowingthe testing team to control time interactively ontheir own simulated production servers.29.8.4 Lessons LearnedThe main thread of this story follows a versionof a software developer’s adage:“Wrap externaldependencies.” While the runtime library isnormally considered internal, the clock itprovides is a service outside the system. Whenthe passage of time is important in system logic,wrapping the clock is a beneficial move.The unexpected bonus was that adaptingto successively larger scopes (isolated code,scripted captive system, interactive deployedsystem) provided benefit to more and differentgroups of people and for different types of tests.Although the larger scopes required modestlymore architectural plumbing, in each case thewrapped clock fit into configuration systemsthat had been built to bring other benefits tothe system. With hindsight, it would have beenbetter to build them earlier had we knownmore of the operational and testing uses for theapplication clock.I’ve now moved on to other work within thecompany, but I can see the application clockhas been maintained and adapted to fit thesystem’s new configuration mechanisms. I’mglad it continues to prove useful.or them.LessonLook for wider application for any useful utilitiesthat help to automate some aspect of testing. 29.9 Flexibility of the Automation SystemMike Bartley, United KingdomLecturer and consultantWe developed our test automation systemourselves and devised ways to adapt ourautomated testing to be more efficient in waysthat we have not seen elsewhere.Because we had already developed an inhousesoftware version control and build system, itwas fairly easy to integrate our automation toolwith our build system. This made our testingmore efficient because we could selectivelytest only those modules that had changed, asshown by our source code dependency tree. Ifnothing that a particular test depended on hadchanged, that test would not be executed. Thisdramatically reduced the build and cycle timeand thus allowed us to put in place continuousintegration of builds and tests. We did keep anoption that forced all tests to rebuild and run ifwe wanted to run a full regression test.
  22. 22. Test Automation Anecdotes21PAGEWe made it easy to remove tests from the testsuite when a test needed updating because ofchanges in the software it was testing. Althoughwewerethencuttingdownonthetestcoveragebecause the test(s) were not run, it meant thatthe maintenance of those tests didn’t have tobe done immediately, thereby stopping the restof the test suite from running.We extended this to a way of“banning”specifictests for various reasons:• The software has changed, but the tests have not yet been updated.• A test is known to be giving a false failure (i.e., it fails but it should pass).• A test is not restoring test machines to a known good state.This idea proved to be a major benefit to ourefficiency.TipBeing able to selectively choose tests to run ornot to run can make the automation quickerand more efficient. Make it clear which tests areactive and which are not. 29.10 A Tale of Too Many Tools (and Not Enough Cross-Department Support)Adrian Smith, United KingdomQA leadI have been involved in five automation projectsover 5 years, with varying degrees of success.29.10.1 Project 1: Simulation Using aDSTLThefirstprojectwaswritteninPythonandbatchscripts, running functional and performancetests on 85 to 145 PCs, simulating more than20,000 machines. It was not originally anautomation project, but I made it so. Whatstarted as a simple control program ended upgrowing into a fully flexible domain-specific testlanguage (DSTL), as it would now be called, thatenabled the tester to write test steps in a simplethough unstructured keyword/parameterlanguage. Expand the tests, alter them, chainthem, and add new elements to the languageas needed. The potential 9-month project wasstill being used 6 years later, It has ended uphaving a much better ROI than expected asits scope has increased over time. Thousandsof man hours were saved and vastly more testruns were performed than a could have beenrun manually and with fewer execution errors.About halfway through the automation project,my manager wanted me to do some manualtesting for a different project because it wasway behind. I knew this would be the end ofthe automation, so I managed to convince himthat it would be better for me to stay with this(ad hoc) project—and this contributed to itssuccess.LessonHave the courage to argue your position withyour manager.29.10.2 Project 2: Testing a GUI UsingTestCompleteThe second automation project was toautomate system-level applications. A toolwas bought to experiment with: TestComplete3. I had high hopes for another success usinga DSTL for the testers, but this would take a
  23. 23. Test Automation Anecdotes22PAGElong lead time to build. We then came acrossproblems of automating GUI interfaces writtenin an automationunfriendly way. I naively askeddevelopment to assist by modifying their codeto help with the automation but was flatlyrefused. I had no support from managementfor this, so I had to go it alone. I probably shouldhave stopped there and then.LessonWithout the cooperation of developers, testautomation will be more difficult than it needsto be.But I didn’t. I persevered with the DSTLframework, though with little abstractionbecause I wanted to have something workingsooner for management. At the time the firstbeta was just about ready, a new director oftesting was appointed.The good news was thathe thought automation was a good thing. Thebad news was that he decided we needed to get“the right tool”with a single solution of manualtest creation, results gathering, and reporting.I had to suspend my work with TestCompleteand was given a 2-month task to evaluate anumber of GUI automation tools.The final threewere Rational Robot, Quality Centre QTP, andguess what:TestComplete. After the evaluation,I thought TestComplete was the most flexibleand wanted to continue with it. The companythought differently, so this framework wasnever completed.29.10.3 Project 3: Rational RobotA 3-month proof-of-concept was then initiatedfor Rational Robot. If I had got further inthe previous project, I could have at leastreused the tests written. It was decided to dosomething similar with this tool, framework,abstraction, and tests being a thin layer ontop. After 8 months, another automator andI had abstracted the tests and had a GUIobject action library that could be generatedautomatically. Many hundreds of lines of codewere automatically generated to do simple GUIactions as click a button or check a textbox. Allthat was changing was which control in whichwindowtouse.Wehadagoodfeelingaboutthisframework because it was simple, and we werejust starting to settle on a project to automatewhen, at this point, management decided to doa proof-of-concept for QuickTest Professional(QTP).29.10.4 Project 4: Final Python Projectand QTP Proof-of-ConceptManagement were now getting involvedand wanted to see some ROI that could bequantified to justify the purchase of this tool.We set to work on a single-use framework inPython, eventually automating 15 end-to-endsmoke tests using GUI libraries. I had made aPython frontend so that testers could createand maintain tests without needing a lot oftechnical knowledge. The number of bugs thatit found was too low to justify extending thisautomation to other areas. However, we werebeginning to get occasional cooperation fromdevelopers. There were a couple of interfacesthat could be called directly from Python to Cwritten specifically for us to enable the tests tofunction.We had one problem that we lost many daystrying to figure out: The tool would crash butdidn’t report any error. It turned out to be a bugin the tool itself.Good PointDon’t forget that the tool itself is also softwareand may have bugs. to be.For the proof-of-concept project for QTP, wehad trouble trying to work with QTP in the waywe wanted to, and a lot of time was wastedcoming to grips with it. Eventually, we found aworkaround to allow us to put many methods
  24. 24. Test Automation Anecdotes23PAGEin one QTP file. At the end of this proof-of-concept, I would still have opted for one of theother tools.Management chose QTP, and we had a realproject to do with deadlines and end dates, somany of our ideals were curtailed, sidelined, ordropped. We again ran into problems with GUIobjects and no help from developers.29.10.5 Project 5: QTP2With a new release of QTP, we tried to integrateour framework and Python code so that testresults would be received centrally while stillallowing us to launch tests including rebootingmachines. This was using VMware virtualizationand CruiseControl. We added extensively toapplication libraries, which were QTP librariesthat did lots of specific tasks in a GUI, passingin a number of parameters. We also wanted tobring the test creation tool up to date so thatthe testers could use the automation easily. Thethought behind this was that the easier it wasto write a test, the quicker it would be to addtests, while the application libraries could bemaintained by the automators.However, management didn’t want“extraneoustime” spent on this perceived nonproductiveactivity!LessonManagers sometimes think they know besteven when they don’t. As an automationchampion, dare to say “Mission Impossible”when necessary.The way we automators wanted it wasthat testers would not have to learn muchprogramming, but because there would be notool to help with creating tests, then testerswould have to learn programming and knowmuch more about the internal workings of theautomation system. This is not a bad thing, butwith a lot of pressure on the test department, itseemed (and was proven to be true) that testersrarely had time to dedicate to automationprogramming before being taken off again byanother project. In a lot of cases, they nevermade it to do any automation because ofdeadline changes. At this time, automationstill was not at the forefront of managementthinking, but it was definitely starting to gettheir notice.Progress was being made, libraries wereexpanding, and the tests were nearingcompletion, having overcome many of theproblems of GUI fragility.Now a problem that we automators had“forgotten about” came back. For the first timein a couple of years, the GUI interfaces beganto be amended or overhauled on a much morefrequent but ad hoc basis (to us automators).We were not informed of changes as theyhappened,soourtestsstartedfailing,andittooka long time to find out why. After 2½ monthsof battling this upset (hundreds of changes toGUIs rendered even the smoke tests useless), Icalled a halt.LessonChanges to the user interface can derailthe automation if it cannot cope with suchchanges.29.10.6 The End of the StoryOf the five automation projects I was involvedin, only the first one achieved success. Itwas non-GUI and an isolated project. Theothers failed because of what seemed to bemanagement decisions and lack of cross-department cooperation, but perhaps bettercommunication would have helped.Management had a bit of a shock and arethink about automation after all the previous
  25. 25. Test Automation Anecdotes24PAGEhigh-profile problems. Automation is nowa deliverable for developers — one of thekey problems before was that there was noincentive for developers or developmentmanagers to support or even cooperate withautomators, as they had their own targets.Direct GUI automation has been abandoned,and automation is now at API level.The final irony is that developers now have tomaintain the APIs and automation code; if onlythey had agreed to maintain object libraries orhad added a few lines to object maps earlier,there would have been less work for themnow. 29.11 A Success with a Surprising EndGeorge Wilkinson, United KingdomTest manager, trainer, and consultantThisanecdotedescribessomeofmyexperiencesonalargetestautomationprojectundertakenin2007and2008.Thisprojectwastoautomatethecore processes of the system validation tests ofthe National Health Service (NHS) Care RecordsSystem (CRS) application as rolled out withinEngland by a large health IT Systems Integrationcompany. This was being undertaken as part ofthe wider National Programme for IT (NPfit).Thestudy covers 8 months of continuous progress,though with a surprising end.An automation team was formed from anumber of locations, including the North ofEngland, the Midlands, and the West Country.Rather than looking for an exact skills match,we wanted people experienced in the CRSapplicationwhowereenthusiasticaboutgettinginvolved in automation. Because the team wasgeographically distributed, we decided to meetmost weeks in a geographically central locationfor 2 days.Good PointTeam-building is important for automationteams, too, especially when they aregeographically dispersed.29.11.1 Our Chosen ToolTestStream was a commercial validation suitefrom Vedant Health, a United States companyspecializing in test health-care automationtargeted at laboratory information systems(LIS) and health informatics systems (HIS). Ourrepresentative from Vedant traveled from theUnited States to start the project going andto run the training in the product set and theTestStream methodology.Good PointTake advantage of learning from others whenthe opportunity arises.One of the useful features of TestStream wascalled Scenario Builder. It provided a way toconstruct automated patient journeys, whichare made up of a number of predefined actions.The analyst simply pulls together these actionstocreatealongertest.Thereareover600actionsforourCRSapplicationsystem,andtheyincludeelements such as Register a Patient, Add Allergy,Schedule Surgery, and Check in a Patient. TheScenario Builder allows the sequence of eventsto be defined and viewed as a true patientjourney.No scripts, scripting, or further scriptdevelopment was required by either my teamorVedantHealth,becausetheScenarioBuilder’sactions provided the components or scenariosrequired. The only requirements were a solidfamiliarity with the application under test and
  26. 26. Test Automation Anecdotes25PAGEa thorough understanding of the test case(normally a patient journey).We built a comprehensive library of automatedscripts and devised standards and proceduresabout how they were to be stored andmaintained. We developed a customizedcomparison and data collection tool, which wecalled CAT (collection analysis tool).29.11.2 The Tool Infrastructure and anInteresting Issue as a ResultThe product was installed and accessibleby the user via a secured network to serversrunning virtual machines (VMs), as shown inFigure 29.3. Access to the VMs and thus to thetest environments was provided to both theautomation team running tests and companyIT support staff.Vedant’s access for support could be fromanywhere in the world because some of thehighly experiencedVedant support staff movedaround the world assisting other clients. Thisrequired remote access to our infrastructure,but we soon discovered that it didn’t work.The system was so secure (in order to preventfraudulentaccessintoanytestenvironmentthatmay hold live patient data) that it prevented theremote access facility from working.Good PointDon’t forget to test your support facility,especially if you have stringent securityrequirements.We resolved the issue by allowing bothcompanies independent access to anothertest system that was clean of any patient data.This solution was foolproof from a securityperspective but provided only limited support,which was to be mitigated by the test systemholding the same application version that themajority of systems were holding in the field.Although the solution was not perfect, becausethe deployments were not always running thesame system version, it was a step in the rightdirection—and one on which we could makeprogress.Looking back, we realized that no feasibilitystudy had been conducted on support, whichcould have prevented the remote access issuefrom arising.29.11 A Success with a Surprising End 559Test environmentaTest environmentbUser access toTestStreamSecure serversusing VMware,running severalTestStream clientsfigurE 29.3 TestStream infrastructureFigure 29.3 TestStream infrastructure
  27. 27. Test Automation Anecdotes26PAGE29.11.3 Going toward RolloutOver the next 3 to 4 months, the team grewfrom 6 to 10, with an additional four part-timesupport members. We produced a catalog ofthe automation tests that were available tothe deployment projects to build their ownscenarios. As we progressed with the pilot, weidentified data and configuration requirementsthat were localized to the individual projects asthey moved away from a standard. This meantthat our current generic approach needed tobe tailored for the deployment-specific testenvironment. What we had done was createda process but lost some sight of our individualcustomer’s requirements.LessonDon’t invest too much effort in designing yourautomation without testing it out in a realsituation.Weranasampleofthedatacollectionandclinicalordering features of the CRS for a particulardeployment. This was a great success becausewe found many defects that were therebyprevented from entering the live environment.We found between 10 and 100 defects on well-built and established test environments andthousands on other environments.We published a report to the stakeholdersshowing how we added value to the currentmanual test approach. We found that wecould automate tests for around 70 percentof the installed CRS functionality and saveapproximately 30 percent of our current testingeffort.Wenowdecidedtoinitiatesomepublicrelationsfor the tool. We scheduled several educationalsessions to explain the program and whatwe had been doing, to give stakeholders theopportunity to ask questions, and to gatherfeedback from the teams working on customersites.LessonDon’t invest too much effort in designing yourautomation without testing it out in a realsituation.I was quite surprised at how many people had avery different interpretation than we did of theproduct set and its intentions and of softwaretest automation itself. Most people’s experiencewith automation test tools is that they requireconstant scripting or maintenance to work.Fortunately, these sessions helped to convincepeoplethatourautomationwasanimprovementon that.We also dispelled some illusions andmisperceptions about automation and setmore realistic expectations.The public relationsmeeting also raised the team’s confidence andgave them some well-deserved recognition.The automation team were elated by theresults from the pilot project and the fact wewere now in the rollout stage. Their confidencewas really growing; after all, they had made itwork. TestStream was out there and making areal difference! We were positioning ourselveswell, and the future, at last, after a good deal ofeffort, was looking more positive.29.11.4 The Unexpected HappensIn late May 2008, after discussing our successso far and the rollout plans, the overall projectwas cancelled because of a breakdown in thecontract with the systems integration company.Therefore, our automation project was alsocancelled. I gathered my team together for thelast team meeting and officially announcedthe cancellation. They had worked extremelyhard, but the automation project was over; allthose many late evenings, weekends, sheerdetermination, and extra miles traveling tomake this work were now history. What aheartbreaking end to what should have been a
  28. 28. Test Automation Anecdotes27PAGEgreat success.Good PointSometimes in spite of a great effort, thingsdon’t go the way we expect because of factorsentirely out of our control. Take what you canfrom the experience to use next time. 29.12 Cooperation Can Overcome Resource LimitationsMichael Albrecht, SwedenTest manager, consultant, and speakerI was in a test team that was testing technicalaspects of banking system processing withoutaccess to a GUI. For this project, we needed notjustdomainknowledgebutmoretechnicalskillsthan we had. Rather than take the traditionalproject approach and try to hire someone,we got everyone together, both testers anddevelopers, and developed the skills we neededbetween us, although we did need to bringsome testers with coding skills into the team.We had no money for tools, so we just startedto use the same open source tools that thedeveloperswereusing.Thedifferenceinthewaywe used them was that we needed to do somecoding to create scenarios for our tests ratherthan just exercising each one individually. Wealso needed to verify our expected results anddoublecheck with the database directly.We didn’t spend any money on tools, but wedid spend a lot of time (we also built our ownperformance-testing tool). Sometimes it iseasier to explain (or hide) these costs: Thepurchase of a tool would appear to be a largesingle cost, but time being spent over monthsor years doesn’t appear to be as significant evenif it is the same or even more money!We found that close cooperation with thecustomer and working as a team enabled us tosucceed in our automation. Being forced to usethe same tools was a blessing in disguise in theend.P.S. from Lisa Crispin: In one project (a goodwhile ago), I knew I needed to get help fromthe developers to progress the automation,so I decided to use the same programminglanguage for the tests that the developers wereusing.Iboughtmyselfabookandstartedwritingscripts. When I needed help, the programmerswere happy to help me because they knew thelanguage.LessonCooperation between testers and developersis good for automation, and so is extendingthe tools you already use. And sometimesdeviousness works! 29.13 An Automation Process for Large-Scale SuccessMichael Snyman, South AfricaTest managerIworkforalargebankinSouthAfrica,employing25,000 staff. We adopted test automation from2006 to 2008 with a clear aim of reducing costsand increasing productivity.
  29. 29. Test Automation Anecdotes28PAGEIt was Edward Deming who said, “If you can’tdescribe what you are doing as a process, youdon’t know what you are doing.”In our case, thiswas true; any success in the area of automationwas due to individual skill and a large amountof luck. The challenge was in taking what thesuccessful individuals did and describing thispractice in the form of a process.29.13.1 Where We StartedOur shelves were littered with numerous toolacquisitions and implementations with varyingdegrees of success. Each of these individualattempts had been focused on very limitedand sometimes selfish objectives. The habitof looking only at accomplishing immediateproject goals had significantly affected theability of the organization to optimally useits selected tools. Such a one-sided view ofautomation had a considerable negative effecton operational activities such as regressiontesting and on justifying the investment made.Compounding the problem was the loss ofvaluable information in the form of test cases,test scenarios, and test data.LessonFocusing on too low a level in the organizationdoes not optimize automation as a whole.Automation was involved too late in theprocess. How often is automation viewed asthe savior of the behind-schedule project?When automation does not deliver on theseunrealistic expectations, it becomes yetanother automation failure. In reality, it is verydifferent; my experience points to automationrequiring multiple cycles and project releasesfor it to become fully effective and provide anacceptable ROI.We weren’t capitalizing on what we could havelearned. For example, a failure experienced inproduction is an example of a test missed andone that should be included in the test casesfor the next release. Test automation shouldprovide an interface for both manual testersand incident management systems with theaim of capturing lessons learned during anyphase in the project lifecycle.The seeming lack of success in test automationinitiatives and the large upfront investmentrequired deters projects from planning andimplementing test automation. The reluctanceto learn from unsuccessful implementationsand the habit of blaming the tool for failure inautomation projects has resulted in a stigmalinked to specific tools and automation ingeneral.Past attempts to justify automation focused onqualityasthekeyattributetobeconsideredandmeasured.Thedifficultyindealingwithqualityisthat it is extremely complex. We clearly neededa way of providing a cost–benefit calculationfor test automation using an attribute otherthan quality.Good PointAppropriate objectives are critical. Automationdoes not improve the quality of the tests or thesoftware being tested.In the absence of a detailed automationframework and process, a large dependencywas placed on the skill and ability of individualteam members.29.13.2 The Key to Our Eventual Success:An Automation ProcessIn 2006, a formal project was launched withdedicatedresources,achampionforautomation,a good technical framework, clear goals, anda detailed plan. In this anecdote, I describeone aspect that was critical to our success inachieving automation on a large scale.
  30. 30. Test Automation Anecdotes29PAGEIt was clear, based on past experience, that astandard approach for automation should bedefined and documented in the form of a testautomationprocess.However,thisprocesscouldnot exist in isolation but had to be integratedinto the newly defined manual test process andshould be compatible with the organizationalsoftware development lifecycle (SDLC). Forexample, in the requirement for a definedautomation process, the framework requiredhigh-level activities described as specificationanalysis,scriptcreation,scenariodocumentation,validation, and data sourcing that needed to besatisfied by a detailed process. The full processis shown in Figure 29.4.TipThe more people who are involved inautomation, the better the documentationabout it needs to be.From the documented automation framework,we were able to extract the keyprocess activities required to perform andsupport most automated testing activities.Here follows a brief description of the objectiveof each step.• Analysis and design: Understand the client’s requirements, and establish if it is possible to satisfy each requirement with current technology at our disposal.• Scripting and configuration: Implement the client’s requirements via an automated solution. This might include recoding, coding, and building special utilities.• Parameter definition: Assess scripts against system user–defined scenarios with the aim of identifying elements to be parameterized.• Parameter management: Manage large amounts of data in customized spreadsheets.• Scenario collection: Populate spreadsheets with scenarios provided by stakeholders of the system.• Validation: Check the spreadsheets and parameters, incorporating pass and fail criteria in the spreadsheets and allowing the automated script to validate results of executed tests.• Testing of scripts: Ensure that the scripts run as expected, and remove any bugs in the scripts.• Script execution: Run the scripts with the scenarios and parameters defined.• Review of results: Internally review the results of script execution, what tests passed and failed, any common problems29.13 An Automation Process for Large-Scale Success 565■ Testing of scripts: Ensure that the scripts run as expected, and remove anyAnalysis anddesignTesting of scripts Validation ScenariocollectionParametermanagementParameterdefinitionScripting andconfigurationScript executionReview ofresultsResultcommunicationfigurE 29.4 Automation processFigure 29.4 Automation process
  31. 31. Test Automation Anecdotes30PAGE such as an unavailable environment, and so on.• Result communication: Summarize the results sent to managers, developers, stakeholders, and others.29.13.3 What We LearnedThese are the main lessons we learned on ourjourney through test automation:• Having a tool is not an automation strategy. a. The tool is nothing more than an enabler of a well-thought-out set of automation activities. b. We believe that if you approach automation correctly, you should be able to switch between tools with little or no impact.• Automation does not test in the same way as manual testers do a. Automation will never replace manual testers. We view automation as an extension of the manual tester, taking care of mundane activities such as regression testing, leaving the tester to get on with the more intellectual work.• Record and playback is only the start. a. A set of recorded, unparameterized scripts has very limited reuse and ages quickly. The focus on data-driven automation provides us with the flexibility and reuse required.• Automation test scripts are software programs and must be treated as such. a. Follow a standard software development life cycle in the creation of automated scripts. b. Document requirements; design, implement, and test your automated scripts.• The value lies in the maintenance. a. The secret of getting a good return on your investment is reuse; for this to be possible, ensure maintenance is simple. b. Keyword or data-driven approach facilitates both reuse and easy maintenance.29.13.4 Return on InvestmentOur automation process enabled us to achieveconsistency of automation practices across thebank. We showed a benefit of $8,600,000 after3 years. This benefit calculation method wasreviewed by our finance team at the highestlevel, and the benefits were confirmed by theindividual system owner for whom the testingwas done.Thetotalamountinvestedinthetestingproject,of which automation was a subproject, was inthe area of $4,200,000. The amount spent onautomationwaslessthan20percentofthistotalbudget, including the acquisition of functionaltesting tools, consulting, and the creation andexecution of automated test scripts.The benefit calculation was primarily based onthe saving achieved in human resource costs.For example, one of our main systems usedin the sales process took, on average, 4 weekswith 20 human resources to regression test.With automation, we reduced that process to 5days and two resources: a reduction from 2,800manhours to 70 man-hours. This translatedto a financial savings of about $120,500 perregression cycle. If you take into account that,on average, we run two full regression cyclesper release and have multiple system releasesper year, and that we are involved in variousother systems, the savings soon start addingup.We have a spreadsheet that uses parametersas the basis for all calculations. It allows usto compare the manual execution time perparameter to the automated time. We refer toparametersastheinputsrequiredbythesystemunder test (e.g., if we are testing a transfer from
  32. 32. Test Automation Anecdotes31PAGEone account to another, parameters might be“from account,” “to account,” and “amount”).So, if we say that conservatively we invested$850,000 in automation and had benefit of$8,600,000, then the ROI for automation (ROI =(Gain − Cost)/Cost) was over 900 percent.From a project testing perspective, theorganization viewed the return on the totalinvestment in testing, which was still over 100percent. (Usually, if ROI is 10 percent or more, itis considered an excellent investment!)It is also interesting to note that the automationpart was the only initiative within the projecttesting that could be measured accurately, andas such, it provided justification for the entireproject.Good PointKeep good records of the costs and benefitsof automation (and testing) to make sure thehighest levels in the organization realize what agoodinvestmenttheyhavemadeinautomatingtesting. 29.14 Test Automation Isn’t Always What It SeemsJulian Harty, United KingdomTester at largeI strongly believe testing can and should useautomation appropriately, and conversely,we should be careful not to waste time andresources on automating garbage (e.g.,ineffective, misguided, or useless tests). Also,we should beware of being beguiled by shinyautomation for its own sake, and over the years,I’vesadlymetmanypeoplewhobelieve,withoutfoundation, that because they have automatedtests, these tests are appropriate or sufficient.One of my self-assigned responsibilities as atest engineer is to challenge these flawed testsand retire as many as practical.Good PointJust because a test is automated doesn’t makeit a good test.This anecdote includes several experiencereports of test automation, both good andbad. Generally, I was directly involved in them,but sometimes the analysis was done by otherteam members.They are taken from companiesI’ve worked with and for over the last 11 years.Project teams ranged from about 10 to 150technical staff and typically ran for severalyears.In every case, test automation was core to theproject.29.14.1 Just Catching Exceptions DoesNot Make It a Good TestA large global application included several APIsthat allowed both internal and external groupsto integrate with it. Java was the primaryprogramming language. Over the years, beforeI was involved, hundreds of automated testshad been written for the respective APIs. Forone API, the tests were written as a separateapplication,startedfromthecommandline,andin the other, the open source JUnit frameworkwas used. Each set of tests ran slowly, andseveral days were required to update thetests after each release from the application’sdevelopment team.Our team of test engineers was asked to assumeresponsibility for both sets of tests. Eachengineer was assigned to one set of tests. We