Best practices for implementing automated functional testing ...


Published on

  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Best practices for implementing automated functional testing ...

  1. 1. Best practices for implementing automated functional testing solutions White paper Table of contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2 Functional testing versus unit testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2 Why automate functional testing processes? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 The benefits of automated functional testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 How to identify candidates for automated testing . . . . . . . . . . . . . . . . . . . . . . . . . . .4 How to determine the ROI of automated testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4 Evaluating automated testing software: What to look for . . . . . . . . . . . . . . . . . . . . . .5 HP Quality Center software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 HP Functional Testing software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 HP Business Process Testing software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10 Checklist: Five keys to successful automated testing . . . . . . . . . . . . . . . . . . . . . . . . .11 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12
  2. 2. Abstract: Today’s enterprises need to conduct thorough This paper presents a brief overview of the advantages functional testing of their mission-critical applications and challenges of automating functional testing to help to verify that all business processes work as expected. structure your thinking about how best to approach By implementing automated functional testing, your test automation. It also summarizes the key features company can dramatically increase both the speed and benefits of HP Functional Testing software—HP and accuracy of your testing processes, providing a QuickTest Professional software, HP WinRunner higher return on investment (ROI) from software projects software, and HP Service Test software—the industry’s while dramatically cutting risk. most popular and powerful automated software testing solutions. The paper concludes with a brief overview of HP Business Process Testing software and presents a checklist of the five keys to successful automated testing. Introduction Functional testing versus unit testing There is no question that rigorous functional testing Functional testing provides the ability to verify that is critical to successful application development. applications work as they should—that they do what The challenge for your developers, QA teams and users expect them to do. Functional tests capture user management alike is how to speed up testing processes requirements in a useful way, give both users and and increase accuracy and completeness—without developers confidence that the business processes breaking your already tight budgets. meet those requirements, and enable QA teams to verify that the software is ready for release. By automating key elements of functional testing, your company can meet aggressive release schedules, test Functional testing is complementary to—but quite more thoroughly and reliably, verify that business different from—unit testing. Simply put, unit tests processes function correctly, and ultimately generate indicate whether the code is doing things right; higher revenue and customer satisfaction from your functional tests tell whether the completed application online operations. is working correctly and providing the proper functionality. Unit tests are written from the code Yet the prospect of automating functional testing raises developer’s perspective, while functional tests are new concerns: written from the end user’s and business-process • What are the costs of automating the testing perspective. processes and what is the ROI? • Which applications/processes are candidates for automated testing and which are not? • Will new training be required and how will that impact current development project schedules? • What is the proper methodology for automating a testing effort? • What is involved in maintaining automated tests? • When comparing test automation products, which features are most important? These and other issues need to be fully examined and understood before test automation projects are initiated. 2
  3. 3. Figure 1: Application testing today Severity Total number Examples More permutations result from higher infrastructure complexity. Platforms 1 Intel Operating systems 5 Windows® XP, ME, 2000, NT4, and 98 Front-end clients 4 Internet Explorer 6, Netscape 7.1, Java, Visual C++ Business processes 5 Login, search, order entry, order confirmation, order fulfillment Data sets 15 Combinations of usernames, passwords, search strings, order numbers, ship dates, etc. Total tests to run 1 x 5 x 4 x 15 + 1,500 possible test scenarios! Why automate functional testing The benefits of automated functional processes? testing The pressure on today’s IT organizations continues to The benefits of functional test automation are many, build. Management is looking to IT to deliver new and include the following: capabilities, unlock new business opportunities, and • Enable faster execution. Computers are orders of deliver competitive advantages through software. This magnitude faster than humans at executing functional means the addition of many more business application test scripts, so more tests can be done in less time, development projects with tighter deadlines—but not more applications can be tested in a given period, always with more budget or incremental resources. and more projects can be delivered on schedule. At the same time, management is becoming increasingly And unlike humans, computers work 24 hours a day, aware of the critical links between software and revenue. including nights, weekends and holidays; they don’t Web services, online transaction processing (OLTP) and get bored or tired; and they don’t make assumptions enterprise resource planning (ERP) applications are not about what works and what doesn’t. only mission-critical, they’re directly tied to your company’s • Facilitate greater test coverage. Automated functional ability to generate revenue. testing products support the execution of test scripts Today’s enterprises rely on very complex computing across all popular browsers, operating systems, etc. infrastructures. As shown in Figure 1, a typical organization Regression testing of ever-changing applications and may depend on multiple applications that were built to environments with automated tools is easier than with work on different operating systems, use several different manual processes. And with integrated data-driven front-end clients, involve numerous business processes spreadsheet capabilities, test automation products and interact with many separate data sets. Testing all allow your developers and QA teams to perform possible permutations of these components creates a calculations, manipulate data sets and quickly create highly complex testing situation with hundreds or multiple test iterations to expand test case coverage. thousands of testing scenarios. With automated testing tools, any mix of transactions When software fails, the expense can be extremely and any user workload can be quickly emulated. high—from lost sales to unproductive employees to • Deliver higher test accuracy and find more defects unhappy customers and demoralized development and earlier. Test automation gives your developers an QA teams. The later in the development cycle defects easier way to replicate and document software are found, the more expensive they are. A defect defects. This helps expedite development processes identified in a production environment can be greater while verifying correct functionality across all than 100 times more expensive to fix than the same environments, data sets and business processes. defect identified earlier in the design phase. Automation is key to improving the speed, accuracy and flexibility of the software testing process, enabling your company to find and fix more defects earlier. 3
  4. 4. • Provide formalized processes. The introduction of How to determine the ROI of automated testing encourages your test teams to formalize their processes, resulting in higher test automated testing consistency and better documentation records. The ROI for any investment can be obtained from • Facilitate the re-use of tests. Once tests are scripted, a simple calculation: your developers can use, re-use and add to the test ROI = Net present value of the investment suite as they make changes to their applications. _______________________________________ There is no need to recreate scripts to test the same Total initial cost functionality for each application modification. • Support ERP/CRM implementations. As more When it comes to automating functional testing companies are implementing ERP/CRM solutions, the processes, the costs are tangible but the net present need to do end-to-end regression testing is becoming value also includes many intangible factors. The best more frequent and more crucial. approach is to determine with as much precision as possible what the hard-dollar costs are and then How to identify candidates for compare them to the hard- and soft-dollar benefits of automated testing automating the test efforts. In general, it makes sense to focus automation efforts Hard-dollar costs to consider in an ROI calculation on critical business processes, complex applications, include: and the use cases that comprise them (as opposed to • Acquisition costs: The cost of initially acquiring the lower-level tasks, such as system-level verifications). test automation software. But if your enterprise has multiple software testers • Hardware costs: The cost of the hardware required spending many hours per day and still has quality and for functional testing. Typically, no special-purpose functional problems, you could definitely benefit from hardware is needed for functional testing, provided moving to automated testing. the tester has access to a standard desktop computer The decision of whether or not to automate the test or workstation with an Ethernet port. process should be driven by ROI considerations. In • Labor costs: Factor in the cost of trained personnel general, there will be a positive return if the application to script test cases or build manual tests, including requires multiple builds/patches/fixes, needs to be the costs of recruiting, hiring, paying and retaining tested on numerous hardware or software configurations, trained experts. and supports many concurrent users. In addition, if repetitive tasks, such as data loading and system configuration are involved, or if the application needs to meet a specific service-level agreement (SLA), automation will certainly make economic sense. 4
  5. 5. • Training costs: Depending on which test automation • Integrated data tables: One of the key benefits of product is selected, the user may need considerable automating functional testing is the ability to pump training to become proficient at writing scripts for large volumes of data through the system quickly. But automated testing. Alternatively, companies may it is also important to be able to manipulate the data elect to hire professional services firms to handle the sets, perform calculations, and quickly create hundreds initial automated test creation. of test iterations and permutations with minimal effort. Your enterprise should look for products that offer When weighing the potential rewards of automation, integrated spreadsheets with powerful calculation it is important to consider intangible benefits, such as capabilities. higher morale and job satisfaction among testers, improved customer satisfaction and loyalty, and an • Clear, concise reporting: The ability to run a high enhanced reputation for reliable software among volume of tests is of little benefit if the results of the end users. tests are not easy to understand or interpret. Testing products should automatically generate reports that Evaluating automated testing display all aspects of the test run and explain the results in an easy-to-read format. The reports should software: what to look for provide specifics about where application failures Many vendors offer test automation products. Every occurred and what test data was used; present solution has its own strengths and weaknesses, unique application screen shots for every step to highlight features, and market niche. Each enterprise’s specific any discrepancies; and provide detailed explanations requirements will determine which option is the best fit. of each checkpoint pass and failure. It should also However, there are several key capabilities that should be possible to share these reports across an entire be included in any test automation product: QA and development team with no modifications. • Scriptless representation of automated tests: The testing product should offer a point-and-click interface for accessing and interacting with the application components under test—as opposed to presenting line after line of scripting. Testers should be able to visualize each step of the business process and view and edit test cases intuitively. This will shorten the learning curve for testers and help QA teams meet urgent deadlines. 5
  6. 6. Figure 2: HP Quality Center software components. Center software HP Quality Center™ Dashboard TestDirector Requirements Defect Extensions Management Test Plan Test Lab Management SAP, SOA Functional Testing Business Process Testing QuickTest Service WinRunner Accelerators for SAP, Oracle, Security Professional test Foundation Shared data repository Central administration Workflows Open APIs HP Quality Center software HP QuickTest Professional software ActiveScreen technology offers the industry’s shortest learning curve HP Quality Center software provides a complete, web- for testers and its TestGuard technology provides easy- based, integrated system for performing QA across a to-maintain tests. It satisfies the testing needs of both wide range of IT and application environments. It technical experts and business analysts, enabling your includes an integrated suite of role-based applications company to deploy higher-quality applications faster, and best practices, as well as an open, scalable and with reduced risks and cost. extensible foundation—all designed to optimize and automate key quality activities. HP Quality Center soft- HP QuickTest Professional software provides support ware components include solutions for requirements, for additional environments and includes the test, defects management and ERP/SOA; functional, following core enhancements: regression and service testing; and business-process • Provides support for HP Business Process Testing design and validation. software to create keyword-driven, component- based tests, allowing subject-matter experts and HP Functional Testing software HP QuickTest Professional software engineers to HP Functional Testing software—HP QuickTest Professional work together software, HP WinRunner software and HP Service Test • Enables testers to create and use Multiple Application software—allows your enterprise to automate functional Areas for each part of the application tests, enabling your IT teams to quickly identify defects • Offers a Function Definition Generator to quickly and verify that applications work correctly prior to document and register new user-defined functions for going live. tests and components Visibility into application readiness • Provides an Open XML report format for test results HP QuickTest Professional software is our advanced • Provides the ability to pass data in tests and components automated testing solution for building functional and using parameters regression test suites. Its innovative, zero-configuration, keyword-driven approach to structured automation • Includes an Object Repository Comparison Tool and provides the ability to use natural language to build Object Repository automation for easier management tests that verify user interactions and determine that • Offers Navigate and Learn capabilities to simplify business processes work as designed. It is the industry’s object repository creation best solution for functional and regression test automation • Provides support for .NET 2.0, web, Java™, web for practically every software application and environ- services, and Windows environments ment. It enables your testers to capture, verify and replay user interactions automatically to identify defects. • Includes an HP QuickTest Professional software script editor tool to edit multiple test scripts • Provides a single license mechanism for HP QuickTest Professional software and add-ins 6
  7. 7. Figure3. Environments supported by HP WinRunner software and HP WinRunner HP QuickTest HP QuickTest Professional software. software Professional Preliminary Classic Common Emerging software Custom C/S Web-related environments ERP/CRM • Powerbuilder • IE, Netscape, AOL • SAP • Forte • JDK, Java™ Foundation • Siebel 7.x • Delphi 7 Classes, AWT • Peoplesoft 8.x • Centura • Activex • Stingray ERP/CRM Custom client server .NET • Smalltalk • Oracle®: 11i • Windows/Win32 • Winforms • JD Edwards • Visual Basic • Webforms ERP/CRM • HTC/Viewlink web client • BAAN Operating systems • Delphi 8 .NET • Peoplesoft • Windows 98, 2000, Winforms Windows NT, ME, XP • Siebel 5, 6 GUI clients Legacy • Oracle GUI forms • 3270, 5250 emulators • VT100 HP WinRunner software Choosing Between HP QuickTest Professional software HP WinRunner software is the industry’s most popular and HP WinRunner software functional testing solution. Currently, more than 24,000 HP WinRunner software and HP QuickTest Professional companies and 250,000 QA professionals use this software are both excellent functional testing solutions. powerful solution. HP WinRunner software captures, The choice of which product to use should be based verifies and replays user interactions automatically, so upon the computing environment in which the testing testers can identify defects and determine that business will be conducted, as well as decisions to leverage processes work flawlessly upon deployment and the enterprise’s existing investments in HP WinRunner remain reliable. HP WinRunner software adds the software scripts and employee training. The environ- following core enhancements ments supported by HP WinRunner software and HP and provides support for additional environments: QuickTest Professional software can be classified into • Delivers a new highly customizable user interface three categories: classic, common and emerging: • Provides a framework-based test design support • Classic: These are environments supported only through Compiled Module functions with a new by HP WinRunner software. This category includes drag-and-drop interface client/server environments, such as PowerBuilder, as well as earlier ERP/CRM applications, such as Siebel • Includes an Advanced Debugger and Dockable 5 and 6. Debug Interface • Common: Includes environments supported by • Presents a new GUI-Map configuration both HP WinRunner software and HP QuickTest • Provides enhanced memory management Professional software, such as web and Windows • Offers updated environment support for Windows (Win32). 2003, the latest Active X, C++, Visual Basic, MFC, • Emerging: This category consists of environments Terminal Emulation, Oracle, web, Java updates and supported by only HP QuickTest Professional PowerBuilder 10 software and includes .NET and newer ERP/CRM environments, such as SAP and Siebel 7. 7
  8. 8. Figure 4. HP WinRunner software Testing Function HP QuickTest Professional software HP WinRunner software and HP QuickTest Professional Active screen Yes software side-by-side feature comparison. Analog Yes Yes Auto-documentation Yes Code editor and debugger Yes Yes Function generator Yes Yes XML object repository Yes Text GUI map Yes Integrated data table Yes Keyword view Yes Numerous checkpoints Yes Yes ODBC and excel connectivity Yes Yes Open multiple tests Yes Open multiple function libraries Yes Yes Rapid object import Yes Yes Record/replay Yes Yes Recovery manager Yes Yes Run wizard Yes Script and function libraries Yes Yes ScriptFusion Yes Shared object repository Yes Yes Strong database integration Yes TestGuard Yes Business process testing Yes Yes TSL Yes VBScript Yes XML reports Yes Environment extensibility Yes Automation object model Yes Missing resources pane Yes HP is committed to providing ongoing support for Our goal is to enable our customers to continue to HP WinRunner software in all classic and common realize the benefits of using their chosen functional environments, and is dedicated to making HP testing solutions in their existing environments as these QuickTest Professional software work in all common environments evolve. We also intend to further increase and emerging environments. For example: the level of integration between HP WinRunner software • When a new version of PowerBuilder is released and HP QuickTest Professional software, and between and gains market acceptance, HP intends to build both of those products with HP TestDirector software. support for that new version of PowerBuilder into HP The goal is to enable direct communication between WinRunner software (classic environments). HP WinRunner software TSL scripts and HP QuickTest Professional software VBScripts. In practice, this means • When a new version of Internet Explorer is released, that development is being undertaken so that both HP intends to build support for it in both HP WinRunner products should be able to call and launch scripts from software and HP QuickTest Professional software one another, and pass parameters and test results to (common environments). each other. We also plan to continue to develop HP • As new versions of .NET, SAP and Siebel are released, TestDirector software as the common management HP intends to support those environments with HP solution for both products. QuickTest Professional software (emerging environments). 8
  9. 9. An optional migration path from HP WinRunner HP Service Test software for functional testing of software to HP QuickTest Professional software SOA services If you have large investments in HP WinRunner Service-oriented architectures (SOAs) provide a set software test assets, or if you need to test applications of services that can be shared and re-used across in classic environments (like PowerBuilder), you can application development initiatives. SOA solutions choose to stay with HP WinRunner software with the enable your organization to: assurance that HP intends to continue to develop and • Coordinate services instead of supporting large, support advancements to the product’s functionality. monolithic applications But if you want to move to HP QuickTest Professional • Decouple applications from underlying infrastructure software you can use HP Functional Testing software to facilitate the transition. • Shift the orientation from technology to business HP Functional Testing software combines HP QuickTest HP Service Test software provides a solution that Professional software and HP WinRunner software into a enables QA engineers to simplify the ongoing testing single integrated product that can leverage test assets from of SOA services and reduce scripting through auto- both HP WinRunner software and HP QuickTest Profes- mation. Users can efficiently enable compliance with sional software scripts. Your quality engineers can use HP corporate standards, even in organizations that use Functional Testing software to create “composite scripts” multiple development tools. HP Service Test software consisting of tests built in both HP WinRunner software does not require a UI for services testing, and the and HP QuickTest Professional software. HP Functional scripts created can be used for both functional and Testing software leverages the integration between HP performance testing. WinRunner software and HP QuickTest Professional HP Service Test software enables organizations to software, and test results are reported into a common conduct functional test and regression test automation, reporting interface. and it addresses every major software application and environment. The solution satisfies the needs of both technical and non-technical users, and enables your IT organization to deploy higher-quality services faster, cheaper and with less risk. IT can rely on HP Service Test software for functional testing and HP LoadRunner software for testing the performance of services. IT has unique, shared needs for both functional and load testing as well as specific needs for each, and HP integrates HP Service Test software and HP LoadRunner so QA can efficiently manage functional and performance testing of services. 9
  10. 10. HP Business Process Testing software The pros and cons of manual functional testing Though the benefits of automated testing are many, In addition to our functional testing solutions just there are some situations where manual testing may described, HP Business Process Testing software provides make sense from a functional or cost perspective. To higher-level, scripted automated functional testing make the decision between automated and manual capabilities to business analysts so that they too can be testing, it is important to understand the strengths and part of the quality process. HP Business Process weaknesses of manual functional testing processes. Testing software works hand-in-hand with HP QuickTest Professional software and HP WinRunner Some of the disadvantages of manual testing include: software to bring non-technical subject-matter experts • Manual tests take too long. Limited IT resources into the quality process early, in a meaningful way. combined with tight delivery schedules make manual Plus, it empowers your entire testing team to create testing simply too time-consuming to be effective to sophisticated test suites with less training. meet business objectives. With manual testing, your HP Business Process Testing software reduces testers and developers have to tediously document the overhead for automated test maintenance and each step of a test case and then manually execute combines test automation and documentation into a each test as well as reproduce defects, rapidly single effort. With this solution, organizations are able consuming valuable time and resources. According to measure the quality of application deliverables to Aberdeen Group, an independent industry analyst, from abstract business definitions defined within 90 percent of all IT projects are delivered late, and the HP Business Process Testing software framework. manual testing is a contributing factor to these delays. Subject matter experts can focus on creating high-level • Manual tests provide incomplete coverage. The test flows that mirror actual business processes, while proliferation of platforms, operating systems, client quality engineers concentrate their efforts on areas devices, business processes and data sets wreaks than enable automation. havoc on manual testing processes. As testers begin HP Business Process Testing software also includes: to combine multiple data sets, operating environments • Pure manual components and full manual HP and business processes, they create an explosion in Business Process Testing software execution the number of test cases that need to be executed to verify the functionality of an application. This makes • Permissions and access control enforcement it very difficult to perform verifications of all of the • Easy automation infrastructure for HP Business multiple permutations. And, when development fixes Process Testing software generation in HP QuickTest are introduced, manual regression testing takes too Professional software, with: long to make complete regression tests practical. — An application area manager — A function library for maintenance and debugging — Built-in function definition generator in HP QuickTest Professional software • An API for creating/managing business components as a “technology preview” 10
  11. 11. • Manual tests introduce higher risk. Manual testing Checklist: five keys to successful processes are subject to a higher risk of mistakes and oversights than computer-driven processes. People automated testing get tired, they make errors entering data, they don’t Even when there is clear evidence that automating a always code the tests correctly, and they don’t testing effort is economically justified, it can be difficult always have the time to test everything that should to determine how best to approach the transition to be tested. automated testing processes. This section outlines five But for some environments, manual testing may be an fundamental principles for implementing automated appropriate choice. For example, manual testing may software testing processes: be appropriate for applications under heavy development, 1.Complete a test plan document. Understanding the that frequently change their UI or logic, applications goals of the application to be tested is critical to the built on top of non-supported technologies or homegrown success of any testing effort. This includes thorough UI toolkits, or lower-level tasks, such as system-level up-front planning to verify that test requirements are verifications. implemented correctly. HP TestDirector software Benefits of manual testing include: provides the ability to manage both test cases and requirements for all applications to be tested. • Manual tests can provide fast access to testing information. Testing can be started before the 2.Subdivide the tests into test cases to be automated. application is fully stable, providing faster feedback It is impossible for your organization to automate all to the development team. aspects of a test plan. Automated testing should be focused around the complex and critical business • Manual tests provide the ability to perform processes that map to functionality within an application simultaneous testing. Manual testing can be done as designed per the requirements. Many organizations simultaneously with the development and the find that they are automating around 60 percent of debugging of automated scripts. their total number of test cases, leaving 40 percent • Manual tests scripts can be easy to create. Manual of tests to be conducted manually. testing scripts are easy to create by business users. These steps could be later converted to automated scripts. • Manual tests are easy to understand. Manual testing scripts are easy to understand by all users, and can be provided to business users for test validation and to developers as requirements documents or steps that lead to bugs. 11
  12. 12. 3.Create automated tests. HP WinRunner software Summary Data-Driver Wizard greatly simplifies the process of Functional testing need not be a time-consuming or preparing test data and scripts. This allows for better expensive proposition. By automating functional testing, use of QA resources and results in more thorough your enterprise can take major steps forward in your testing. Using HP QuickTest Professional software, ability to improve automated business processes. Your users can create tests without having to do any actual development and QA teams can increase both the scripting. HP QuickTest Professional software captures speed and accuracy of the testing processes, and your the business process for the target application itself IT department can achieve a higher ROI from software and allows users to create test flows that can then be projects while cutting risk. saved into HP TestDirector software and managed via the web. By selecting HP WinRunner software, HP QuickTest 4.Do data-drive tests to expand test coverage. Using Professional software or HP Service Test software as the built-in data-table feature of HP QuickTest the test automation platform, your enterprise can take Professional software, your testers have the ability advantage of the most sophisticated yet easy-to-use to create data-dependent tests that use specific functional testing technology solutions on the market. keywords stored in Excel spreadsheets to populate fields in an application. This allows your testers to drive massive volumes of test data through an appli- cation. (Note: HP WinRunner software does not have a built-in data table, but does offer all of the func- tionality described above.) 5.Add verifications to the tests. The actual pass or fail criteria for a test are then added to the HP QuickTest Professional software or HP WinRunner software tests. This includes verifications of the front- end of the application, the middle tier or the back- end database. Built-in database verification confirms the values stored in the database and verifies trans- action accuracy and the data integrity of records that have been updated, deleted or added. To learn more about HP functional testing solutions, visit: © Copyright 2006, 2007 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Microsoft, Windows and Windows NT are U.S. registered trademarks of Microsoft Corporation. Oracle is a registered U.S. trademark of Oracle Corporation, Redwood City, California. Java is a U.S. trademark of Sun Microsystems, Inc. 4AA1-1390ENW, April 2007