8 Steps to Better Test Automation


Published on

Automation doesn't automatically make software testing faster, more reliable and less expensive. Up-front costs in automation tools and set-up can be high. In addition, not all automation tools and methodologies have the same features, and each project may have different requirements.

To help you get the maximum benefit from automated testing, Cognizant offers eight tips based on our experience with more than 50 enterprise-level automated testing projects, worldwide. Following these tips can help increase the ROI of test automation and improve software quality.

Published in: Technology
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

8 Steps to Better Test Automation

  1. 1. Cognizant White Paper Eight Steps to Better Test Automation Automation doesn’t automatically make software testing faster, more reliable and less expensive. Because the up-front costs in automation tools and set-up can be high, automated testing only pays off if the long-term cost savings offsets those initial expenses. In addition, not all automation tools and methodologies have the same features, functions and capabilities, and each project may have different requirements that affect its costs and benefits. To help you get the maximum benefit from automated testing, we offer eight tips based on our experience with more than 50 enterprise-level automated testing projects, worldwide. Following these tips can help increase the ROI of test automation and improve software quality. Top Eight ways to achieve efficient test automation. whitepaper
  2. 2. One: Choose What to Automate such as Windows and Linux, but often cannot do the same for controls Not all software projects lend themselves created by development tools equally to testing. Understanding which vendors such as Infragistics, and factors increase the difficulty of automating application vendors such as Oracle a specific test project is essential to and Siebel. balancing the costs against the benefits. • Code or test cases for which the More appropriate candidates for automated production or infrastructure teams have not supplied the right type of testing include code that: data, or the proper amount of data, to support a full-scale automated • Plays an important role in an test program. application • Code which changes often, since • Processes large amounts of data such changes may also require time- • Executes common processes consuming manual revisions to either • Can be used across applications the test script or the test data. We have developed a nine-point decision tree (see Nine-Point Test Decision Tree Figure 1) that helps clients select automation candidates based on the following criteria: • Technical feasibility • The frequency of test execution • The extent to which test components can be reused • Total resource requirements • The complexity of the test cases • The ability to use the same test cases in multiple browsers or environments • The time required to execute the tests Two: Choose Your Test Tools Because companies must amortize their investment in automation, they should select Figure 1 automated test tools that will meet their needs for years. Proper evaluation criteria include: • Support for various types of automated Organizations should also look for non- testing, including functional, test traditional and even unplanned areas, over management, mobile, SOA and IVR which they can spread their automation (interactive voice response) software. investment. These include automating the • Support for multiple testing frameworks testing of installation routines for patches and (the methodologies that enable the re- defect fixes, automating test management, usability and efficiency of automation). and the creation of test reports. • The ability to recognize objects created in a wide variety of programming Less appropriate factors for automated testing languages. include: • Stable setup and operation on any environment/platform. • The extensive use of non-standard • Ease of debugging the automation software controls (the components scripts. that define the user interface). • Efficient test execution with a minimum Automated test tools typically can of manual effort. assess the quality of standard • Automatic recovery from application controls within operating systems 2 whitepaper
  3. 3. failures to prevent test interruptions. the required licenses are purchased. A powerful scripting language that Among the open-source tools we have makes it easy to develop scripts (the customized are the Selenium suite for instructions carried out in a specific automated testing of Web applications and test) that can be reused across the Bugzilla defect tracking tool. We have platforms and test types. also created a framework based on the WATIR • The ability to report results, and be (Web Application Testing in Ruby) toolkit used controlled by, a wide variety of test to automate browser-based tests during Web management platforms. application development. • Ease of use to minimize training costs and delays. Tool Selection There are several commercial tools that provide these capabilities, but they are expensive. The decision to procure them may solely depend on the proposed investment, and as mentioned above, after considering future requirements. Customizing such open- source tools to meet specific testing needs can be an effective way of reducing costs. Another way to speed ROI from automated testing is to use inexpensive software tools that, while not designed for that purpose, can be useful in automating certain scenarios. Figure 2 Examples: • Use of the macro-recording Three: Refine Your Test Process capabilities that ship with most operating systems to automate In many organizations, the lack of centralized, processes such as loading test data, standardized automation processes has led to an scheduling, and executing tests. elongated testing lifecycle that is prohibitively • Free or low-cost file comparison expensive and fails to detect the maximum utilities can be used to determine number of defects. Improving such processes whether actual test output matches requires the following (see Figure 3): the expected results at far lower cost than using commercial off-the- • Describing the risks of current shelf test automation tools. methods, and showing how testing can be done at a lower cost, more We have developed a methodology to help quickly and/or more effectively. customers choose the test automation software • Management commitment to provide most appropriate to their needs, and to make the funding and support for changing the most effective use of both new and existing workflows required to improve automated test tools (see Figure 2). It begins testing processes. with defining the objectives for the tools and • Avail support from the test staff for specifying the tests to be automated, such as meeting improvement goals. functional testing or back-end validation. The • Training to provide specialized skills next step is to define the requirements, build to test managers, specialists and an evaluation scorecard, perform a proof testing engineers in testing of concept, and finally prepare the tools for methodologies. deployment. This methodology also helps • Prioritizing process improvements customers optimize their use of automated based on business goals. test tools by identifying all testing needs • Ongoing measurement of test across the organization, creating an inventory processes to achieve higher return of tools already available, and reviewing on investment. existing licensing agreements to ensure only whitepaper 3
  4. 4. A Standardized Test Automation Framework A Hybrid Approach to Test Automation Figure 3 Four: Choose a Framework Just as with any tool, automation test solutions must be used correctly to Figure 4 be effective. Choosing an appropriate framework can help boost long-term It allows multiple test cases to be driven with reusability and efficiency. multiple sets of data, and parameterizes the data to ease maintenance and modification. A framework is not a replacement for It drives efficiency by allowing business an automation tool, but a roadmap for users to design and change tests without its effective use. It should also allow for learning complex scripting languages. The parameterization (separate storage) of test use of keywords makes it easy to measure scripts and test data to allow the maximum how much of the code required to perform reuse and easy maintainability of both. In common business functions, such as this way, when changes are made to a test “validate applicant’s age,” is reused over case, only the test case file needs to be time. updated, while the driver and start-up scripts remain the same. Similarly, the test data can Five: Don’t Underestimate the Manual be changed as needed without requiring Effort changes to the test scripts. The word “automation” implies that machines Among the popular frameworks are those do the work, not humans. But the amount that are “data driven,” where the test data of manual effort required in “automated” is stored separately from the automation testing is one of the least understood aspects tool. This provides significant ease of use of software testing. Humans must set up the and customization of reports, eases data test machines, create the scripts, select the maintainability, and allows multiple test tests, analyze the test results, log the defects cases to be conducted with multiple sets and clean up the test machines. An accurate of input data. However, the initial cost and estimate of these costs is important not only maintenance overhead can be considerable. for budgeting and planning, but also to build Another approach is “keyword-driven” an accurate return on investment calculation. in which data tables and keywords are maintained independently of the automation In our client work, we have identified the tool and the test scripts that drive the tests. factors IT organizations should take into But this is somewhat harder to use than the account in estimating the manual effort data-driven approach, which increases costs required for test automation. These include and delays. the complexity of the language used to create test scripts, and the amount of We have developed a hybrid framework (see work required to plan, generate, execute Figure 4, above) ,that combines the best and maintain the scripts. Another aid to elements of both the keyword- and the data- estimating effort is classifying test cases as driven approaches. It stores the test data simple, medium and complex based on the apart from the automation tool, (usually in number of transactions and the number of an Excel spreadsheet), making it very easy to steps within the scripts required by each maintain and reuse the scripts. case. 4 whitepaper
  5. 5. Six: Watch Those Scripts Seven: Identify Who Will Execute the Tests Areas in which standards are particularly im- Automation raises the expectation among portant include the following (see Figure 5): managers that testing can be achieved with • Exception handling, which determines little or no manual effort. Thus, they do not whether, and how well, the test script allocate the staff required to perform the can recover from the failure of the manual steps required in automated testing, application being tested, or unexpected such as analyzing test results and creating behavior such as the appearance of a and cleaning up test machines. Many of the pop-up. The use of standards helps likely candidates for such work are either ensure all scripts continue to run in unavailable or lack the proper skills. the event of unexpected conditions, reducing the need for manual For example: intervention and speeding test execution. • Functional testers already have their • Error logging, in which standards time allocated to manual testing. make it easier for multiple developers and • Automation engineers, while qualified testers to analyze and act on the to run the tests, typically don’t test results. understand enough of the functional • Documentation standards in areas requirements to analyze the failed such as comments and indentation scenarios. They are too busy help developers create uniform scripts, developing and maintaining tests to and to understand script code created execute the tests and track defects. by others. • Without planning for the proper numbers of skilled staff, an automated testing program will fall victim to Scripting for Success unexpected outages, delays and cost overruns. The organization will not be able to run enough tests frequently enough to justify its investment in test automation. That’s why it is important to determine who will own, and execute all the processes around the actual automated testing at the beginning of the process. As an experienced test automation consulting firm, we can help customers make sure they have the right amount of skilled staff to cost-effectively en- sure software quality. This includes accurate- ly assessing the manual requirements at all stages in the automatic testing process, and Figure 5 making better use of existing staff. Functional testers can be retrained, for example, to both Along with the development and use of execute and maintain scripts (see Figure 6): scripting standards, we have developed consolidated toolsets for script creation that have saved customers 10% to 15% in script- ing costs by allowing greater script reuse. whitepaper 5
  6. 6. Resourcing the Test Automation Function testing. • Reduced time to market. • Improvements in customer satisfaction. • Increased productivity due to improved application quality. Figure 6 All these are real benefits that must be balanced against staffing and tool licensing Eight: Measure Your Success -- Accurately costs to determine an accurate return on investment. Tools such as the Automated Execution Efficiency Ratio (AEER) can determine how Measuring what Matters effectively you are executing automated tests by analyzing the ratio of human effort as a percentage of the total effort needed to execute the automated tests. But don’t just compare the number of person-hours required for manual vs. automated tests for a given size of the code. Be sure to include other benefits such as the following (see Figure 7): • A higher percent of defects found. • Reductions in the time needed for Figure 7 Real-World Example: A global financial services company turned to Cognizant to speed its testing of business-critical code, and to improve the return on its automation investment through the reuse of test components. Cognizant helped the client establish a central function to perform automated functional and regression testing of key modules such as trading, payments, accounts, safekeeping and wealth management, and to implement an automation architecture. Cognizant reduced the overall time required for testing by 72%, and the effort required to design and execute multilingual tests by 70% by creating a test quality center and developing a framework to maximize component reuse and the ROI of automated testing. The client has automated 43% of its regression testing, reuses 60% of its automation scripts, and has extended the benefits of this work by implementing the automation framework on multiple projects. Cognizant also developed a dashboard that provides business and technical managers measurable performance metrics through dashboards showing weekly progress on key performance indicators and service level agreements. Conclusion: Automating software testing is not as simple or as quick as the term implies. Software test tools can be expensive, and setting up, executing and analyzing test results requires extensive manual work. But by using the proper tools such as open-source software, and the proper processes and automation frameworks, organizations can realize the cost savings and quality of automated software testing. About Cognizant: Cognizant (NASDAQ: CTSH) is a leading provider of information technology, consulting, and business process outsourcing services. Cognizant’s single-minded passion is to dedicate our global technology and innovation know-how, our industry expertise and worldwide resources to working together with clients to make their businesses stronger. With over 50 global delivery centers and more than 68,000 employees as of September 30, 2009, we combine a unique onsite/offshore delivery model infused by a distinct culture of customer satisfaction. A member of the NASDAQ-100 Index and S&P 500 Index, Cognizant is a Forbes Global 2000 company and a member of the Fortune 1000 and is ranked among the top information technology companies in BusinessWeek’s Hot Growth and Top 50 Performers listings. Visit us online at www.cognizant.com.