Software Test Management Tools Defined


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Software Test Management Tools Defined

  1. 1. Software Test Management Tools Defined by Rick Riccetti Testing software applications can be a daunting task that requires hundreds to thousands of unique test cases, the time to execute them, and the ability to efficiently manage the results. As software becomes increasingly complex and development schedules more aggressive, teams are frequently overwhelmed by the sheer magnitude of the testing effort. Although many teams stumble through the testing process group of tests needed to validate related requirements, or special without task-specific tools (you can always use a spreadsheet purpose testing like stress or user acceptance testing. for basic tracking of test cases and results), task-specific tools • A test plan organizes test cases and test suites to meet testing increase productivity by bringing order and efficiency to an objectives. Test plans define test cases, test suites, the schedule, inherently complicated process. In this paper we’ll discuss the team responsibilities, deliverables, testing tools, success criteria, challenges of managing the testing process and how a new class and tracking procedures. of software applications, called software test management tools, will help QA teams manage testing more efficiently, estimate Why bother with a test plan? Because testing without a test plan test coverage more accurately, and improve the QA process and is a recipe for disaster. A carefully thought-out test plan: resulting product quality. • Ensures all areas to test are adequately tested. • Lays out the most efficient way to structure the testing process. Test for Success • Aids in estimating the testing effort so you know where you are The need to test software is obvious. Testing is the only way to and when the testing process will be complete. verify that all features, including functionality, speed, scalability, • Allows the workload to be distributed while ensuring testers are and security, have been implemented to specifications. The goal not wasting time testing the same things. is to ensure end-to-end traceability of the development process, from requirements through design, to implementation and finally, Almost everyone agrees they must test software somehow. verification. Testing also verifies that the application doesn’t crash, How or how well varies greatly depending on the organization’s: data cannot be lost or corrupted, and the application meets • Culture—is the organization process oriented or less structured? company standards. • Resources—does the organization have the right people, time, and tools to design and implement test plans? The daily activities of software testers typically revolve around the • Knowledge—do team members have the appropriate education design, implementation, and execution of test cases, test suites, and or experience in software testing? test plans. • External pressures—are compliance audits a possibility? • A test case is the basic building block of a test plan. Test cases Winging it through software testing was never the right path to quality define the set of conditions or variables under which a tester software and the complexity of today’s applications further increases determines if a requirement is partially or fully satisfied. It is the risk of proceeding without a plan. Senior management is more a set of activities, supporting data, and expected results. aware than ever of the cost of fixing bugs and dealing with negative • A test suite is a collection of test cases that are grouped together customer feedback after a product ships or is internally deployed. as dictated by the test plan objectives. Test suites can be used to This increased visibility puts QA managers under additional pressure specify which tests should be run at specific project phases, the to improve quality, while continuing to hit release dates.
  2. 2. A Focus on Quality • Lower costs and shorten delivery times: Delivery times shrink The increased focus on software quality has resulted in a greater and costs fall because software test management tools help need for tools that help manage the testing process. Tools have balance the testing effort among team members. Software long added efficiency and traceability to the quality process. engineers are provided with the steps and data they need to Consider the substantial productivity benefits offered by defect recreate a bug, giving them a better chance of fixing the problem the first time and reducing the need to wait on a tester for tracking software. Automated software testing tools, although not information. Software test management tools also simplify the yet as popular, have also provided meaningful benefits. move to automated testing by storing manual and automated Tools that help automate planning and management of software tests in one tool and tracking which tests have been automated testing are the next frontier in quality assurance products. Such and which remain manual. software test management tools will manage all facets of the • Meet compliance requirements: Software test management testing process including test case creation, review, scheduling, tools help satisfy regulatory and customer–dictated compliance execution, measurement, resolution, and reporting. QA managers requirements by maintaining detailed histories of test cases, will know what has been tested, what hasn’t, and how much time data, and results. Compliance needs can be direct (e.g., 21 CFR it will take to get the job done. The bottom line is that software test Title 11, Sarbanes-Oxley) or indirect (e.g., third-party audits). management tools will allow QA teams to test more in less time, Test processes, test data, and the results of test runs are stored while achieving a measurable improvement in product quality. securely in a software test management tool and cannot be accessed by unauthorized personnel. With this tool, a manager Automation to the Rescue can easily produce detailed histories of test cases, data, and results to demonstrate a repeatable, accurate testing process. Software test management tools can help organizations improve schedule predictability, increase product quality and job satisfaction, Not All Automation is the Same lower development costs, shorten delivery times, and meet Just to be clear, the automation of test case management is compliance requirements. Let’s take a look at how software test completely different from the automation of the test cases management tools can deliver improvements in each of these areas. themselves. There are many tools on the market to help automate • Improve schedule predictability: The unpredictability of the execution of test cases (e.g., Seapine’s QA Wizard Pro, software projects hampers efforts to determine product release HP/Mercury QuickTest Pro, Rational Functional Tester). Software schedules. A software test management tool tracks which tests test management tools, by contrast, offer value earlier in the testing have been executed and how much testing remains, allowing process by bringing organization to the management of test cases, schedules to be determined far more reliably. It may also track the steps of which may then be automated or executed manually. test execution times so QA managers can know how much time is required to test an application. This historical information can be used to better estimate future testing efforts, costs, and Built for the Task delivery dates. To provide meaningful benefits to QA managers, software test management tools must, at a minimum, include support for the • Increase product quality and job satisfaction: Software test following test assets: test cases, test runs, test variants, and test management tools improve product quality by ensuring all test sets. Workflow automation and comprehensive reporting are also steps are executed, and in the same order, for more consistent essential features of these tools. testing. New defects can be turned into test cases to ensure old problems do not resurface in future releases. Test failures Tools that track generic data types, such as issue management can be quickly pushed into the defect management workflow, solutions, can track and manage some of these assets, but do not which eliminates time consuming, error prone data entry and offer the asset-specific functionality required to completely manage streamlines the QA > Fix > Re-test process. Product quality the testing process (e.g., scheduling test cases from test runs). improves because managers can measure the effectiveness Software test management tools are designed to handle the unique of tester performance. Knowing how to best schedule available attributes and workflows of the test assets. Each of these assets resources frees up senior test engineers to create additional tests requires data fields and processing rules, including versioning in cases, manage the testing effort, or test an application’s more some cases, specific to their application in the testing process. An complex features. examination of the major test assets and their role in the testing process follows.
  3. 3. Test Cases Run 1: March 4, 2007 A test case is a set of conditions, data, actions, expected results, and other criteria used to determine if a product component is TEST PASS FAIL working correctly and meets its specified requirements. A test Test Case 1 3 case is often referred to as ‘the design of a test’. This concept is Test Case 2 3 important when automating the testing process because the design Test Case 3 3 defines the expectations of the test scripts. Test Case 4 X Each test case should provide specific information for testing a Test Case 5 3 product component including: Test Case 6 3 • The purpose or objective of the test Test Case 7 X • The scope of the test • The tested components • The test configuration requirements Test Variants • The set up required before performing the test Test variants are attributes of the tested application that are used to • The constraints used to generate test runs for the test case generate test runs. Variants reduce the number of unique test cases • The steps for performing an action in the tested application that need to be written by using a single test case to test one or • The expected results of each step performed more variable conditions. For example, a cross-platform application • Any files required for performing the test, such as automated needs to be tested on all supported operating systems. However, scripts or test data it would be inefficient to write identical test cases for each target • Any other information you want to track operating system. Using a test variant, the tester writes a single test case and defines a list of target operating systems as a test variant. When generating test runs, the tester selects the variants relevant to Test Case 1 the test case, resulting in a unique test for each test variant value. 2 Case Test Purpose rver Test Case 1 oject se test case veri es the functionality of the Send Email This to a pr n login button on the Edit Task dialog. se ca Purpose a user Purpo case veri es that This test case veri es the functionality of the Send Email button on the Edit Defect dialog. st This te eb. Steps Steps 1 From the main menu, chose View -> Defects W 1ogin.htm the main menu, chose View -> Tasks From via the 2 Select a defect, click Edit /l /web2 Select a task, click Edit 3 Click Send Email alhost 4 Fill in all required/relevant information r ps browse SteStart your Web wing URL: http:/d/loc 5 Click Send 3 Click Send Email Expected Results sswor 1 Defect List Window Opens 1 follo ter the me/pa 4 Fill in all required/relevant information 2 Edit Defect dialog opens, Verify there is a ‘Send Email’ 2 En r the userna button at the bottom of the dialog 3 Send Email dialog opens 3 En te n 5 Click Send 5 Check the email account to verify email was sent k Logi 4 Clic esults s logged Expected Results e now cted R og appear - you ar Expeogram Login dialdialog will appear 1 Task List Window Opens 2 Pr ction ect sele 2 Edit Task dialog opens, Verify there is a ‘Send Email’ button 4 Proj the server. at the bottom of the dialog into 3 Send Email dialog opens 5 Check the email account to verify email was sent Windows XP Windows Vista Mac OS X Solaris Test Runs A test run is a snapshot of a test case that is used to perform the test and enter the results. Test runs must include all of the Test Sets information from the test case so a tester can correctly set up Deciding which test case needs to be run at which point in the the testing environment and perform the test. During the test, development/test cycle is a central component to a good test plan. the tester can document any results that are different from the Test sets group test cases and test runs and provide a way to track expected test case results. When the test is complete, the tester testing progress against the test plan. Test sets are also an effective passes or fails the test run to indicate the results. A single test case means of organizing suites of regression tests. may have more than one related test run depending on how many times the test run is generated from the test case.
  4. 4. • What percent of tests are automated? • What parts of the test suite should be automated for maximum return on my automation investment? Test Set In addition, managers should be able to easily generate reports required by external agencies or key customers auditing a company’s quality assurance process. Ultimately though, quality assurance comes down to balancing risk, time, and Test Case Test Case Test Case Test Case Test Case cost parameters. The reporting capabilities of a software test 1 2 3 4 5 management tool should help management measure the risk and tune the quality assurance process to minimize the risk. Measure Twice, Test Once Beyond the Test Assets Software test management tools are poised to have a positive Workflow automation and reporting are also important features of impact in software development organizations looking for ways a software test management tool. These features improve process to measure and improve quality. Most companies have deployed efficiency and help managers track results. configuration management and defect tracking solutions to Workflow automation is an important component of software test manage code and test results, and these solutions have resulted in management tools, with each asset requiring a relatively unique a measurable improvement at those organizations. Software test process. Take test runs for example. When test runs are created, management tools represent a new frontier in improving quality, individual test cases within the run may have an initial state of while protecting delivery times and costs. Waiting (as in waiting to be executed), once started they are In Think of the positive impact that software test management tools Progress, and when completed they are either Passed, Failed, On would have on your organization. Whether your testing team Hold, or the results are Unclear. Workflow automation helps guide is large or small, software test management tools can provide the quality process from start to finish, ensures teams are never many benefits including improved schedule predictability, higher left wondering what to do next, and provides management with product quality, increased job satisfaction, lower development visibility into the testing status. costs, and shorter delivery times. Measurability is critical to process improvement, making easy access to data an important component of software test management tools. Reporting features should be able to help managers answer the following questions: • Do we have sufficient test coverage? • How long will it take to run a test set? • Who are the most effective and least effective testers? About the Author Rick Riccetti is a 25-year veteran of the software development industry. He co-founded Seapine Software, Inc. in 1995 and was the principal developer of Seapine’s flagship product, TestTrack. 5412 Courseview Dr., Suite 200 TEL 513-754-1655 ©2007 Seapine Software, Inc. TestTrack Pro, TestTrack, Surround SCM, QA Wizard, SoloBug, SoloSubmit, Seapine CM, Seapine SQA, and the Seapine logo are trademarks of Seapine Software, Inc. All other company products and company names are either trademarks Mason, OH 45040 FAX 513-754-1660 or registered trademarks of their respective companies. All rights reserved worldwide. Information presented here is accurate as of the time of printing, but is subject to change or revision without notice. 7076.6 Whitepaper Software Test Management.indd 5/07