of Automated Testing


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

of Automated Testing

  1. 1. Avoiding the Pitfalls of Automated Testing by Allan McNaughton Q uality assurance engineers are being asked to take on more complex QA Wizard’s simplicity also significantly reduces long-term maintenance costs. This projects than ever before. Application complexity has increased to is a major benefit as the only certainty the point that formerly simple test plans now encompass hundreds, if not in automated testing is that test scripts thousands, of test cases that must be executed under stringent deadlines. eventually break. Tests are created using These demands have resulted in quality assurance teams seeking higher assumptions about the application under test. When the application changes in a efficiencies than traditional manual testing provides. meaningful way it is quite likely scripts Much of the focus of this search has been the selected tool be a good fit with your will fail. Users of language-based tools on automated testing tools. It is commonly software technologies (MFC, .NET, Web, will find themselves laboriously poring thought that automated testing is the etc.) and your testing team’s technical over lengthy scripts trying to rectify the panacea for all quality assurance woes. But is capabilities. problem. Users of QA Wizard can see at a that always the case? While a well-executed glance why tests failed and can update out- Automated testing tools basically fall into automated testing strategy can measurably of-date scripts without touching a single two camps: those that rely on scripting improve productivity, in reality there are line of code. languages and those that use a graphical many pitfalls waiting to ensnare the unwary. approach. Code-based tools have complex Plan for Success Quality assurance teams need to carefully scripting languages but can be customized Automated testing projects must be analyze the path before them. By choosing in enumerable ways. While this flexibility carefully planned to ensure success. The the right tool for their needs and using it may seem desirable, it can cause more first step is to set realistic goals. Many wisely, they can reap the much heralded problems than it solves. These tools are testers think the goal should be to automate benefits of automated software testing. extremely complex, and they can be 100% of the test plan. Experience has Select the Right Tool daunting to all but the most technically shown that just reaching the 75% level is Nothing will sink an automated software sophisticated users. This complexity often a noteworthy accomplishment. The fact is testing project faster than adopting the results in the tool getting shelved after the that manual testing is still required even in wrong tool for the job. Testers are faced initial enthusiasm fades. the most heavily automated environments. with a bewildering array of choices that A better solution is to seek out a tool that However, by using QA Wizard you can range from loosely supported open source does not require programming skills yet reach higher levels of automation faster solutions to commercial tools that claim offers functionality comparable to those that than what is possible with language-based to solve most every automated testing do. Seapine Software’s QA Wizard fulfills tools. problem. this requirement by combining the power Determining which parts of the application Which tool is right for the job? The of a language-based tool with an easy-to- are right for automated testing and which question is best answered by honestly use graphical interface. This dramatically parts should continue to be tested manually evaluating your organization’s needs against reduces the time it takes to learn the tool is straightforward when the following the products that are available in today’s and lets all team members participate in guidelines are heeded: competitive marketplace. To succeed at automated testing projects, regardless of automated testing, it is imperative that their technical skills. 1
  2. 2. ■ Identify portions of the application that projects. Projects exist within workspaces require data comparison or validation. and multiple workspaces can be created. Text comparison and data validation Some teams create one workspace while are areas that manual testing can easily others create separate workspaces for each overlook. QA Wizard can compare and application being tested. Likewise, some validate the same information with total teams create projects for each component in accuracy. an application while others create projects ■ Identify which parts of the application based on the type of testing taking place deal with complex business rules. (See Figure 1). Although QA Wizard can easily direct Think Small the flow of a script using context checks Develop short maintainable scripts. Longer and conditional statements, it is generally scripts are harder to debug and less reusable. more effective to manually test complex With QA Wizard, you can record an business rules. individual script for each test in the plan ■ Avoid automating areas of the application or easily break down a longer script into that change constantly. Automated multiple smaller scripts after it has been testing is better suited for more stable recorded. components (such as core functionality) The process of script decomposition that can reuse test scripts. highlights a key difference between QA Adopt Best Practices Wizard and code-based tools. In a code- As with any tool there are many ways to based tool you must locate the code that approach a problem. To get the most out of implements the test, cleanly extract it, and automated testing it is recommended that create a new script that seamlessly integrates a number of easy-to-follow best practices this logic. As any tester knows, it is all too be followed. These simple steps will help easy to make a mistake when doing this type you reap significant rewards by streamlining of task. QA Wizard visually presents what the creation of automated test scripts and each step does, not how it is accomplished. reducing maintenance of those scripts down This makes moving a step from one script to the road. another a simple matter of copying the step to its new location (See Figure 2). Organize Your Process Ad hoc scripting should not be permitted. Every test script should fit into an overall architecture. Figure 1 Project Explorer The importance of properly ■ Identify components of the application organizing test scripts rises in that require large amounts of data entry. direct proportion to the number Data entry is time consuming and can be of tests that must be automated. easily automated. QA Wizard can retrieve When there are hundreds of these data values from a spreadsheet or automated tests it is easy to be database and enter them for you. overwhelmed without a well- defined system for managing ■ Identify areas of the application that them. require repetitive testing with different data values for each iteration. QA QA Wizard scales to support the Wizard can retrieve these values from a largest test suites by allowing spreadsheet or database and run through users to organize test scripts the test multiple times using new data hierarchically into workspaces and Figure 2 Step copy and paste values with each run.
  3. 3. Plan for Reuse SOAP servers or to access an operating Create reusable scripts to test common system service such as reading and writing user interface components. For instance, system registry keys. Assemblies can also be an application often contains multiple used for custom data manipulation such as occurrences of drop-down menus. While generating unique login names. Methods drop-downs always work the same, testers can be called directly from QA Wizard so usually implement separate test logic for individual script steps can be customized as each one. A more reusable implementation needed. is to develop a callable function that Capture Knowledge determines the number of items in the Test scripts should be documented so they menu and iterates through them one by one Figure 3 Creating checkpoints can be easily understood by others. QA until all choices have been tested. Wizard’s graphical approach to automated After each test run QA Wizard generates a Validate Results testing relieves testers of much of this work. report that shows whether a checkpoint or While automated testing tools are Every step of a QA Wizard test script has any other step in a test script failed. Testers commonly used to control an application as an image showing exactly what field the can quickly ascertain the nature of failures if it were being used by a manual tester, they script is testing (See Figure 4). This visual by examining the actual and expected results may not catch all the errors that a person documentation can be quickly understood of the script. Analysis is further simplified can unless additional measures are taken. by those who are new to a project. Testers as QA Wizard captures actual screenshots of For example, validating a spreadsheet may can further clarify scripts by inserting failing script steps as they occur. require the tester to enter specific values into script steps that only contain descriptive a worksheet and then verify the results of a Use Data Driven Scripts comments. calculation. If the script does not validate Test plans often require that the same test results, the test may pass even though the be executed using different parameters application generated an incorrect answer. (user names, account codes, etc.). The Testers can add validation steps to their work required to accomplish this can test scripts by creating checkpoints using be dramatically reduced by having QA QA Wizard’s point-and-click interface Wizard retrieve the test inputs from a (See Figure 3). QA Wizard supports the data file. When a script is run, the input following types of checkpoints: information is read from the external data file. Subsequent iterations of the script use ■ Value checkpoints verify that an object Figure 4 Previewer a fresh set of records from the file. Data can contains a certain value such as a button be retrieved from a spreadsheet, database, or Harness Integration label or the result of a calculation. multitude of other data sources. Automated testing tools are a key part of the ■ Attribute checkpoints verify that an object has a specific attribute or property Test Deeply software development process. To facilitate Thoroughly testing an application requires tester productivity they should integrate such as class or HTML ID. more than just clicking buttons using an with source code control (SCC) and defect ■ Format checkpoints verify that an object tracking tools. automated testing tool. For example, if an has a specific format or property such as application writes to the Windows registry QA Wizard makes life easier for test position, font, or text alignment. during execution it would be useful to know developers with its support for SCC ■ Contains checkpoints verify that a parent that the correct value was written. This tools such as Seapine’s Surround SCM object contains specific child objects. requires leaving the confines of the testing and Microsoft Visual SourceSafe. This ■ Child count checkpoints verify that a tool and delving into the operating system integration allows testers to benefit from parent object contains a specific number to retrieve the right value. file versioning just as programmers do. Files of child objects such as a drop-down list QA Wizard bridges this gap with its support can be checked in and out of repositories with 10 items. for .NET assemblies. .NET assemblies directly from QA Wizard. Testers should can be used to integrate QA Wizard with take full advantage of the robust branching ■ Focus checkpoints verify that the focus is external systems such as email servers or capabilities of Surround SCM. Branches set to a specific object. 3
  4. 4. tests fail or to allow the user the right tool is used intelligently. To reach to examine individual test this goal it is imperative that ease of use failures and then submit be seriously considered when selecting an reports at the push of a automated testing tool. QA Wizard lets you button. This integration rapidly create powerful scripts that can be further drives the development easily understood by testers at all levels of process as TestTrack Pro technical skill. This simplicity lowers costs can send email notifications and enhances productivity by requiring less the moment new issues are training and fewer resources than language- received. This greatly increases based tools. communication between the quality assurance and development teams. Figure 5 Script management in Surround SCM Developers can let you manage multiple versions of a test resolve problems found by QA suite so scripts for today’s version of an Wizard more quickly than normal, application can coexist with those for the as each issue filed in TestTrack Pro next release (See Figure 5). If an incorrect contains the details necessary to script is checked in, Surround lets you duplicate the defect and a screen correct it by rolling back the file to the last shot of the application at time of known good version. failure (See Figure 6). QA Wizard also saves testers the drudgery of manually creating bug reports. It is fully Enhance Productivity integrated with TestTrack Pro, Seapine’s issue Automated testing holds Figure 6 TestTrack Pro integration management system, and can be configured great promise. The tyranny of to automatically submit bug reports as repetitiveness can be broken when Seapine Software, Inc. Headquarters 5412 Courseview Dr., Suite 200 Mason, OH 45040 Toll-free: (888) 683-6456 Seapine Software International Canisiusgasse 6/39 1090 Vienna, Austria About the Author Tel: +43 676 930 63 93 Allan McNaughton is a patent holding technologist and veteran writer with more than fifteen years of industry experience. He is the president of Technical Insight, LLC, a firm specializing in the composition of high-technology white papers. Mr. McNaughton is a frequent Email: info@seapine.com contributor to leading technology publications and can be reached at allan@technical-insight.com. Web: www.seapine.com Seapine Software ™ Changing the World of Software Development 4