Software Test Management Tools DefinedDocument Transcript
Software Test Management Tools Defined
by Rick Riccetti
Testing software applications can be a daunting task that requires hundreds to thousands of unique test cases, the
time to execute them, and the ability to efficiently manage the results. As software becomes increasingly complex and
development schedules more aggressive, teams are frequently overwhelmed by the sheer magnitude of the testing effort.
Although many teams stumble through the testing process group of tests needed to validate related requirements, or special
without task-specific tools (you can always use a spreadsheet purpose testing like stress or user acceptance testing.
for basic tracking of test cases and results), task-specific tools
• A test plan organizes test cases and test suites to meet testing
increase productivity by bringing order and efficiency to an
objectives. Test plans define test cases, test suites, the schedule,
inherently complicated process. In this paper we’ll discuss the
team responsibilities, deliverables, testing tools, success criteria,
challenges of managing the testing process and how a new class
and tracking procedures.
of software applications, called software test management tools,
will help QA teams manage testing more efficiently, estimate Why bother with a test plan? Because testing without a test plan
test coverage more accurately, and improve the QA process and is a recipe for disaster. A carefully thought-out test plan:
resulting product quality.
• Ensures all areas to test are adequately tested.
• Lays out the most efficient way to structure the testing process.
Test for Success • Aids in estimating the testing effort so you know where you are
The need to test software is obvious. Testing is the only way to and when the testing process will be complete.
verify that all features, including functionality, speed, scalability, • Allows the workload to be distributed while ensuring testers are
and security, have been implemented to specifications. The goal not wasting time testing the same things.
is to ensure end-to-end traceability of the development process,
from requirements through design, to implementation and finally, Almost everyone agrees they must test software somehow.
verification. Testing also verifies that the application doesn’t crash, How or how well varies greatly depending on the organization’s:
data cannot be lost or corrupted, and the application meets • Culture—is the organization process oriented or less structured?
company standards. • Resources—does the organization have the right people, time,
and tools to design and implement test plans?
The daily activities of software testers typically revolve around the • Knowledge—do team members have the appropriate education
design, implementation, and execution of test cases, test suites, and or experience in software testing?
test plans. • External pressures—are compliance audits a possibility?
• A test case is the basic building block of a test plan. Test cases Winging it through software testing was never the right path to quality
define the set of conditions or variables under which a tester software and the complexity of today’s applications further increases
determines if a requirement is partially or fully satisfied. It is the risk of proceeding without a plan. Senior management is more
a set of activities, supporting data, and expected results.
aware than ever of the cost of fixing bugs and dealing with negative
• A test suite is a collection of test cases that are grouped together customer feedback after a product ships or is internally deployed.
as dictated by the test plan objectives. Test suites can be used to This increased visibility puts QA managers under additional pressure
specify which tests should be run at specific project phases, the to improve quality, while continuing to hit release dates.
A Focus on Quality • Lower costs and shorten delivery times: Delivery times shrink
The increased focus on software quality has resulted in a greater and costs fall because software test management tools help
need for tools that help manage the testing process. Tools have balance the testing effort among team members. Software
long added efficiency and traceability to the quality process. engineers are provided with the steps and data they need to
Consider the substantial productivity benefits offered by defect recreate a bug, giving them a better chance of fixing the problem
the first time and reducing the need to wait on a tester for
tracking software. Automated software testing tools, although not
information. Software test management tools also simplify the
yet as popular, have also provided meaningful benefits.
move to automated testing by storing manual and automated
Tools that help automate planning and management of software tests in one tool and tracking which tests have been automated
testing are the next frontier in quality assurance products. Such and which remain manual.
software test management tools will manage all facets of the • Meet compliance requirements: Software test management
testing process including test case creation, review, scheduling, tools help satisfy regulatory and customer–dictated compliance
execution, measurement, resolution, and reporting. QA managers requirements by maintaining detailed histories of test cases,
will know what has been tested, what hasn’t, and how much time data, and results. Compliance needs can be direct (e.g., 21 CFR
it will take to get the job done. The bottom line is that software test Title 11, Sarbanes-Oxley) or indirect (e.g., third-party audits).
management tools will allow QA teams to test more in less time, Test processes, test data, and the results of test runs are stored
while achieving a measurable improvement in product quality. securely in a software test management tool and cannot be
accessed by unauthorized personnel. With this tool, a manager
Automation to the Rescue can easily produce detailed histories of test cases, data, and
results to demonstrate a repeatable, accurate testing process.
Software test management tools can help organizations improve
schedule predictability, increase product quality and job satisfaction,
Not All Automation is the Same
lower development costs, shorten delivery times, and meet
Just to be clear, the automation of test case management is
compliance requirements. Let’s take a look at how software test
completely different from the automation of the test cases
management tools can deliver improvements in each of these areas.
themselves. There are many tools on the market to help automate
• Improve schedule predictability: The unpredictability of the execution of test cases (e.g., Seapine’s QA Wizard Pro,
software projects hampers efforts to determine product release HP/Mercury QuickTest Pro, Rational Functional Tester). Software
schedules. A software test management tool tracks which tests test management tools, by contrast, offer value earlier in the testing
have been executed and how much testing remains, allowing process by bringing organization to the management of test cases,
schedules to be determined far more reliably. It may also track the steps of which may then be automated or executed manually.
test execution times so QA managers can know how much time
is required to test an application. This historical information can
be used to better estimate future testing efforts, costs, and Built for the Task
delivery dates. To provide meaningful benefits to QA managers, software test
management tools must, at a minimum, include support for the
• Increase product quality and job satisfaction: Software test following test assets: test cases, test runs, test variants, and test
management tools improve product quality by ensuring all test sets. Workflow automation and comprehensive reporting are also
steps are executed, and in the same order, for more consistent essential features of these tools.
testing. New defects can be turned into test cases to ensure
old problems do not resurface in future releases. Test failures Tools that track generic data types, such as issue management
can be quickly pushed into the defect management workflow, solutions, can track and manage some of these assets, but do not
which eliminates time consuming, error prone data entry and offer the asset-specific functionality required to completely manage
streamlines the QA > Fix > Re-test process. Product quality the testing process (e.g., scheduling test cases from test runs).
improves because managers can measure the effectiveness
Software test management tools are designed to handle the unique
of tester performance. Knowing how to best schedule available
attributes and workflows of the test assets. Each of these assets
resources frees up senior test engineers to create additional tests
requires data fields and processing rules, including versioning in
cases, manage the testing effort, or test an application’s more
some cases, specific to their application in the testing process. An
examination of the major test assets and their role in the testing
Run 1: March 4, 2007
A test case is a set of conditions, data, actions, expected results,
and other criteria used to determine if a product component is TEST PASS FAIL
working correctly and meets its specified requirements. A test Test Case 1 3
case is often referred to as ‘the design of a test’. This concept is Test Case 2 3
important when automating the testing process because the design
Test Case 3 3
defines the expectations of the test scripts.
Test Case 4 X
Each test case should provide specific information for testing a Test Case 5 3
product component including: Test Case 6 3
• The purpose or objective of the test Test Case 7 X
• The scope of the test
• The tested components
• The test configuration requirements Test Variants
• The set up required before performing the test Test variants are attributes of the tested application that are used to
• The constraints used to generate test runs for the test case generate test runs. Variants reduce the number of unique test cases
• The steps for performing an action in the tested application that need to be written by using a single test case to test one or
• The expected results of each step performed more variable conditions. For example, a cross-platform application
• Any files required for performing the test, such as automated needs to be tested on all supported operating systems. However,
scripts or test data it would be inefficient to write identical test cases for each target
• Any other information you want to track operating system. Using a test variant, the tester writes a single test
case and defines a list of target operating systems as a test variant.
When generating test runs, the tester selects the variants relevant to
Test Case 1 the test case, resulting in a unique test for each test variant value.
rver Test Case 1
oject se test case veri es the functionality of the Send Email
to a pr
n login button on the Edit Task dialog.
se ca Purpose
Purpo case veri es that
This test case veri es the functionality of the Send Email
button on the Edit Defect dialog.
This te eb.
1 From the main menu, chose View -> Defects
W 1ogin.htm the main menu, chose View -> Tasks
2 Select a defect, click Edit
/web2 Select a task, click Edit
3 Click Send Email
4 Fill in all required/relevant information
SteStart your Web wing URL: http:/d/loc
5 Click Send
3 Click Send Email Expected Results
1 Defect List Window Opens
ter the me/pa 4 Fill in all required/relevant information
2 Edit Defect dialog opens, Verify there is a ‘Send Email’
2 En r the userna
button at the bottom of the dialog
3 Send Email dialog opens
5 Click Send 5 Check the email account to verify email was sent
esults s logged
cted R og appear - you ar
Expeogram Login dialdialog will appear 1 Task List Window Opens
2 Pr ction
ect sele 2 Edit Task dialog opens, Verify there is a ‘Send Email’ button
4 Proj the server. at the bottom of the dialog
3 Send Email dialog opens
5 Check the email account to verify email was sent
Windows XP Windows Vista Mac OS X Solaris
A test run is a snapshot of a test case that is used to perform
the test and enter the results. Test runs must include all of the Test Sets
information from the test case so a tester can correctly set up Deciding which test case needs to be run at which point in the
the testing environment and perform the test. During the test, development/test cycle is a central component to a good test plan.
the tester can document any results that are different from the Test sets group test cases and test runs and provide a way to track
expected test case results. When the test is complete, the tester testing progress against the test plan. Test sets are also an effective
passes or fails the test run to indicate the results. A single test case means of organizing suites of regression tests.
may have more than one related test run depending on how many
times the test run is generated from the test case.