Avoiding the Pitfalls
of Automated Testing
by Allan McNaughton
Q uality assurance engineers are being asked to take on more complex QA Wizard’s simplicity also signiﬁcantly
reduces long-term maintenance costs. This
projects than ever before. Application complexity has increased to
is a major beneﬁt as the only certainty
the point that formerly simple test plans now encompass hundreds, if not in automated testing is that test scripts
thousands, of test cases that must be executed under stringent deadlines. eventually break. Tests are created using
These demands have resulted in quality assurance teams seeking higher assumptions about the application under
test. When the application changes in a
efﬁciencies than traditional manual testing provides.
meaningful way it is quite likely scripts
Much of the focus of this search has been the selected tool be a good ﬁt with your will fail. Users of language-based tools
on automated testing tools. It is commonly software technologies (MFC, .NET, Web, will ﬁnd themselves laboriously poring
thought that automated testing is the etc.) and your testing team’s technical over lengthy scripts trying to rectify the
panacea for all quality assurance woes. But is capabilities. problem. Users of QA Wizard can see at a
that always the case? While a well-executed glance why tests failed and can update out-
Automated testing tools basically fall into
automated testing strategy can measurably of-date scripts without touching a single
two camps: those that rely on scripting
improve productivity, in reality there are line of code.
languages and those that use a graphical
many pitfalls waiting to ensnare the unwary. approach. Code-based tools have complex Plan for Success
Quality assurance teams need to carefully scripting languages but can be customized Automated testing projects must be
analyze the path before them. By choosing in enumerable ways. While this ﬂexibility carefully planned to ensure success. The
the right tool for their needs and using it may seem desirable, it can cause more ﬁrst step is to set realistic goals. Many
wisely, they can reap the much heralded problems than it solves. These tools are testers think the goal should be to automate
beneﬁts of automated software testing. extremely complex, and they can be 100% of the test plan. Experience has
Select the Right Tool daunting to all but the most technically shown that just reaching the 75% level is
Nothing will sink an automated software sophisticated users. This complexity often a noteworthy accomplishment. The fact is
testing project faster than adopting the results in the tool getting shelved after the that manual testing is still required even in
wrong tool for the job. Testers are faced initial enthusiasm fades. the most heavily automated environments.
with a bewildering array of choices that A better solution is to seek out a tool that However, by using QA Wizard you can
range from loosely supported open source does not require programming skills yet reach higher levels of automation faster
solutions to commercial tools that claim offers functionality comparable to those that than what is possible with language-based
to solve most every automated testing do. Seapine Software’s QA Wizard fulﬁlls tools.
problem. this requirement by combining the power Determining which parts of the application
Which tool is right for the job? The of a language-based tool with an easy-to- are right for automated testing and which
question is best answered by honestly use graphical interface. This dramatically parts should continue to be tested manually
evaluating your organization’s needs against reduces the time it takes to learn the tool is straightforward when the following
the products that are available in today’s and lets all team members participate in guidelines are heeded:
competitive marketplace. To succeed at automated testing projects, regardless of
automated testing, it is imperative that their technical skills.
■ Identify portions of the application that projects. Projects exist within workspaces
require data comparison or validation. and multiple workspaces can be created.
Text comparison and data validation Some teams create one workspace while
are areas that manual testing can easily others create separate workspaces for each
overlook. QA Wizard can compare and application being tested. Likewise, some
validate the same information with total teams create projects for each component in
accuracy. an application while others create projects
■ Identify which parts of the application based on the type of testing taking place
deal with complex business rules. (See Figure 1).
Although QA Wizard can easily direct Think Small
the ﬂow of a script using context checks Develop short maintainable scripts. Longer
and conditional statements, it is generally scripts are harder to debug and less reusable.
more effective to manually test complex With QA Wizard, you can record an
business rules. individual script for each test in the plan
■ Avoid automating areas of the application or easily break down a longer script into
that change constantly. Automated multiple smaller scripts after it has been
testing is better suited for more stable recorded.
components (such as core functionality) The process of script decomposition
that can reuse test scripts. highlights a key difference between QA
Adopt Best Practices Wizard and code-based tools. In a code-
As with any tool there are many ways to based tool you must locate the code that
approach a problem. To get the most out of implements the test, cleanly extract it, and
automated testing it is recommended that create a new script that seamlessly integrates
a number of easy-to-follow best practices this logic. As any tester knows, it is all too
be followed. These simple steps will help easy to make a mistake when doing this type
you reap signiﬁcant rewards by streamlining of task. QA Wizard visually presents what
the creation of automated test scripts and each step does, not how it is accomplished.
reducing maintenance of those scripts down This makes moving a step from one script to
the road. another a simple matter of copying the step
to its new location (See Figure 2).
Organize Your Process
Ad hoc scripting should not be
permitted. Every test script should
ﬁt into an overall architecture.
Figure 1 Project Explorer
The importance of properly
■ Identify components of the application organizing test scripts rises in
that require large amounts of data entry. direct proportion to the number
Data entry is time consuming and can be of tests that must be automated.
easily automated. QA Wizard can retrieve When there are hundreds of
these data values from a spreadsheet or automated tests it is easy to be
database and enter them for you. overwhelmed without a well-
deﬁned system for managing
■ Identify areas of the application that
require repetitive testing with different
data values for each iteration. QA QA Wizard scales to support the
Wizard can retrieve these values from a largest test suites by allowing
spreadsheet or database and run through users to organize test scripts
the test multiple times using new data hierarchically into workspaces and
Figure 2 Step copy and paste
values with each run.
Plan for Reuse SOAP servers or to access an operating
Create reusable scripts to test common system service such as reading and writing
user interface components. For instance, system registry keys. Assemblies can also be
an application often contains multiple used for custom data manipulation such as
occurrences of drop-down menus. While generating unique login names. Methods
drop-downs always work the same, testers can be called directly from QA Wizard so
usually implement separate test logic for individual script steps can be customized as
each one. A more reusable implementation needed.
is to develop a callable function that Capture Knowledge
determines the number of items in the
Test scripts should be documented so they
menu and iterates through them one by one
Figure 3 Creating checkpoints can be easily understood by others. QA
until all choices have been tested.
Wizard’s graphical approach to automated
After each test run QA Wizard generates a
Validate Results testing relieves testers of much of this work.
report that shows whether a checkpoint or
While automated testing tools are Every step of a QA Wizard test script has
any other step in a test script failed. Testers
commonly used to control an application as an image showing exactly what ﬁeld the
can quickly ascertain the nature of failures
if it were being used by a manual tester, they script is testing (See Figure 4). This visual
by examining the actual and expected results
may not catch all the errors that a person documentation can be quickly understood
of the script. Analysis is further simpliﬁed
can unless additional measures are taken. by those who are new to a project. Testers
as QA Wizard captures actual screenshots of
For example, validating a spreadsheet may can further clarify scripts by inserting
failing script steps as they occur.
require the tester to enter speciﬁc values into script steps that only contain descriptive
a worksheet and then verify the results of a Use Data Driven Scripts comments.
calculation. If the script does not validate Test plans often require that the same test
results, the test may pass even though the be executed using different parameters
application generated an incorrect answer. (user names, account codes, etc.). The
Testers can add validation steps to their work required to accomplish this can
test scripts by creating checkpoints using be dramatically reduced by having QA
QA Wizard’s point-and-click interface Wizard retrieve the test inputs from a
(See Figure 3). QA Wizard supports the data ﬁle. When a script is run, the input
following types of checkpoints: information is read from the external data
ﬁle. Subsequent iterations of the script use
■ Value checkpoints verify that an object Figure 4 Previewer
a fresh set of records from the ﬁle. Data can
contains a certain value such as a button
be retrieved from a spreadsheet, database, or Harness Integration
label or the result of a calculation.
multitude of other data sources.
Automated testing tools are a key part of the
■ Attribute checkpoints verify that an
object has a speciﬁc attribute or property
Test Deeply software development process. To facilitate
Thoroughly testing an application requires tester productivity they should integrate
such as class or HTML ID.
more than just clicking buttons using an with source code control (SCC) and defect
■ Format checkpoints verify that an object tracking tools.
automated testing tool. For example, if an
has a speciﬁc format or property such as
application writes to the Windows registry QA Wizard makes life easier for test
position, font, or text alignment.
during execution it would be useful to know developers with its support for SCC
■ Contains checkpoints verify that a parent that the correct value was written. This tools such as Seapine’s Surround SCM
object contains speciﬁc child objects. requires leaving the conﬁnes of the testing and Microsoft Visual SourceSafe. This
■ Child count checkpoints verify that a tool and delving into the operating system integration allows testers to beneﬁt from
parent object contains a speciﬁc number to retrieve the right value. ﬁle versioning just as programmers do. Files
of child objects such as a drop-down list QA Wizard bridges this gap with its support can be checked in and out of repositories
with 10 items. for .NET assemblies. .NET assemblies directly from QA Wizard. Testers should
can be used to integrate QA Wizard with take full advantage of the robust branching
■ Focus checkpoints verify that the focus is
external systems such as email servers or capabilities of Surround SCM. Branches
set to a speciﬁc object.
tests fail or to allow the user the right tool is used intelligently. To reach
to examine individual test this goal it is imperative that ease of use
failures and then submit be seriously considered when selecting an
reports at the push of a automated testing tool. QA Wizard lets you
button. This integration rapidly create powerful scripts that can be
further drives the development easily understood by testers at all levels of
process as TestTrack Pro technical skill. This simplicity lowers costs
can send email notiﬁcations and enhances productivity by requiring less
the moment new issues are training and fewer resources than language-
received. This greatly increases based tools.
communication between the
quality assurance and
Figure 5 Script management in Surround SCM
let you manage multiple versions of a test
resolve problems found by QA
suite so scripts for today’s version of an
Wizard more quickly than normal,
application can coexist with those for the
as each issue ﬁled in TestTrack Pro
next release (See Figure 5). If an incorrect
contains the details necessary to
script is checked in, Surround lets you
duplicate the defect and a screen
correct it by rolling back the ﬁle to the last
shot of the application at time of
known good version.
failure (See Figure 6).
QA Wizard also saves testers the drudgery
of manually creating bug reports. It is fully Enhance Productivity
integrated with TestTrack Pro, Seapine’s issue Automated testing holds
Figure 6 TestTrack Pro integration
management system, and can be conﬁgured great promise. The tyranny of
to automatically submit bug reports as repetitiveness can be broken when
Seapine Software, Inc.
5412 Courseview Dr., Suite 200
Mason, OH 45040
Toll-free: (888) 683-6456
Seapine Software International
1090 Vienna, Austria
About the Author Tel: +43 676 930 63 93
Allan McNaughton is a patent holding technologist and veteran writer with more than ﬁfteen years of industry experience. He is the
president of Technical Insight, LLC, a ﬁrm specializing in the composition of high-technology white papers. Mr. McNaughton is a frequent Email: firstname.lastname@example.org
contributor to leading technology publications and can be reached at email@example.com. Web: www.seapine.com
Seapine Software ™
Changing the World of Software Development 4