Navigating the Minefield of Open Source Test Tools
Navigating the Minefield of
Open Source Test Tools
By Jeff Jewell
For STAREAST 2005
The process of testing software products can be greatly improved by the use of good
tools. There are tools to help manage test plans and test cases, to automate regression and
load testing, to track defects, and many other testing activities. Software quality can be
increased, and the amount of time it takes to assure that quality can be reduced, by the
implementation and proper use of software testing tools.
However, selecting a software testing tool can often be an arduous process, but not
usually because of a limited supply of available tools. The difficult part of selecting a
tool is the dizzying number of available tools, and the wide variation in the costs and
functionality of those tools. The real challenge becomes finding an affordable tool that
can meet the complex needs of your development environment and products.
The Testing Tools Landscape
The landscape of software testing tools is dominated by a few high-end vendors with
suites of tools that are very rich in features and relatively easy to use. These are usually
the most well-known products, with advertising and sales forces pushing them into
targeted organizations. Their high licensing costs (usually thousands of dollars for a
single seat) put these tools out of reach for a large number of software development
Struggling for light below the canopy of high-end tools are a number of lower cost
products. These tools are typically not as feature laden as the high-end tools, but have
much lower price tags. The mid-range tools tend to focus more narrowly on a specific
market segment, such as automating only web applications and not Windows. Ease of
use may also be an issue, but not necessarily.
Those still not satisfied by what they find in the underbrush of mid-range test tools can
emerge from the forest of commercial products to see a field populated with open source
software. These tools appear to be a dream come true to cash-strapped development
organizations. Most of them require no license fees, and even come with the source code
so you can make your own enhancements. And there are dozens to choose from, touting
advanced functionality and favorable comparisons to high-end products.
With visions of saving their companies thousands of dollars, software test teams rush
headlong into a minefield of open source testing tools, only to discover some unexpected
obstacles. The hidden costs and risks of using open source software can quickly erase the
dreams they once had of a free software testing panacea. However, equipped with the
right information, savvy test teams can find some very useful products they can afford.
This paper will provide some of the information needed to successfully navigate the
minefield of open source test tools. Some of the benefits and common misconceptions of
open source software will be presented, along with information on what it takes to
successfully use open source tools, how to find an appropriate tool, and what challenges
may be faced in its implementation.
What is Open Source Software?
Open source software has publicly distributed source code, and is usually available for
free. Having the source code freely available allows many developers to contribute to the
development and enhancement of the product. Indeed, most open source products are
developed by people who volunteer their time and efforts.
Not all open source software is free, and not all free software is open source. There are a
number of products that are distributed for free without the source code, and are usually
known as “freeware”. Microsoft’s Internet Explorer is a good example of a freeware
product. Microsoft does not publish the source code, but the software can be downloaded
and used for free. “Shareware” products are also often available for free, but have some
mechanism established to encourage or require users to eventually pay for the use of the
product. On the other hand, you may be charged for some open source products, but the
source code will be included. For more detailed definitions of open source software refer
to the Open Source Definition at the Open Source Initiative.1
Open Source Acceptance
The use of open source software is increasing, and a number of products have been
accepted into the mainstream. Two of these very popular open source products have
emerged as the poster children for the open source community. Apache web server now
powers more than 60% of web sites2, clearly dominating a key market segment. The
Linux operating system is also making significant gains in the marketplace, especially for
systems where reliability is essential.
Another open source product that is making waves in the software industry is Firefox, the
latest web browser to be released based on the Mozilla code base. The 1.0 version of
Firefox, which was released in November 2004, has been steadily gaining on the
Microsoft Internet Explorer monopoly:
“In the one-month period from Nov. 5, 2004 to Dec. 3, 2004, Firefox’s online
usage share grew from 3.03 percent to 4.06 percent….
“Firefox has made significant gains and Microsoft’s Internet Explorer’s usage
share continues to fall in the United States. Between Nov. 5 and Dec. 3, Internet
Explorer’s share dropped 1.09 percentage points to 91.80 percent, according to
WebSideStory. This compares with a 0.32 percentage point drop the previous month.
Since June 4, 2004, when IE’s usage share first began to drop due to security
concerns, IE has lost a total of 3.68 percentage points.”3
Not only is Firefox usage growing, but corporate support for it is also increasing. Google
recently hired one of the lead developers of Firefox, and committed 50% of his time to its
The Lure of Open Source
The biggest benefit of open source software is that it’s usually free. Although there are
some costs associated with using open source tools (which will be discussed later), the
fact that you don’t have to pay for licenses is a big draw for budget-conscious companies.
Many people also like the fact that the source code for the software is openly published.
Having access to the source code provides a number of advantages. First, it can give
users a better understanding of how the software works internally. It also lets users
customize the product, including adding new functionality. Finally, if there are problems
with the software, users can fix work-stopping issues without having to wait for an
update from other sources.
Open source software products are distributed via the Internet, and are typically
developed in small iterative cycles. Therefore, smaller and more frequent updates can be
released than with most commercial products. Faster releases can mean that users won’t
wait as long for enhancements and expected fixes.
Finally, the collaborative nature of open source development creates a unique
environment for software to evolve. With many people, from various backgrounds,
contributing to the product development there is the opportunity for greater
experimentation and innovation than with commercial products. In addition, most open
source products are born from a real-world need and are developed by the people who
plan to use them. This results in products that very closely match the needs of users, who
may also find that they are able to get better response to suggestions for improvement.
The Open Source Initiative web site offers the following explanation about the benefits of
open source software:
“The basic idea behind open source is very simple: When programmers can read,
redistribute, and modify the source code for a piece of software, the software evolves.
People improve it, people adapt it, people fix bugs. And this can happen at a speed
that, if one is used to the slow pace of conventional software development, seems
“We in the open source community have learned that this rapid evolutionary
process produces better software than the traditional closed model, in which only a
very few programmers can see the source and everybody else must blindly use an
opaque block of bits.”5
How true this statement is really varies from one product to the next. But several features
of open source development do have the potential to produce very good software.
Many feel that the risks associated with open source software outweigh the benefits.
There are a number of risks in open source products that will be discussed later.
However, the perceived lack of quality in free products is not necessarily true of open
source tools. The large community of people contributing to open source projects can
help to overcome that hurdle. The fact that major corporations are willing to use open
source software in mission critical applications is a great testimony to the quality and
stability that these products can attain.
Are You Ready for Open Source Software?
After hearing all of the great reasons for adopting open source software products you may
be anxious to get your hands on some. Before doing so, however, every company needs
to evaluate if they are prepared to handle the investment it will take to be successful with
open source tools. Depending on the people in your organization and the environment
you work in, open source software may or may not be right for you.
Have You Got the Time?
Most major software acquisitions and implementations require an investment of time and
money, and require certain skills. With commercial products the money spent on the
product up front is often the biggest investment, and can shorten the amount of time and
skills needed through training and professional services help. With open source software
the investment of money will be less significant than the amount of time and the skills
that will be required.
Open source software products will frequently require more time to implement and
maintain than commercial products because of a number of factors:
• Installation and configuration processes may be unrefined
• Training is not usually available
• User documentation may be inadequate
• Ease of use may not be a priority of the product
• Frequent releases make it more difficult to stay up-to-date
If your software team is pressed for time more than money then open source software
may not be the best option for you.
Have You Got the Skills?
The other major factor to consider is whether you have the right skills in your
organization to work with open source software. These products are usually created by
developers, for developers, and require more technical aptitude and expertise than their
A large number of open source products have been developed by people with Unix and or
Java backgrounds. As such, their products may present difficulties for people with only
Windows and Microsoft language experience. Many have command line interfaces
instead of a GUI, or require expertise in less-common programming languages.
All of these factors combine to make products that may be very challenging for QA
organizations. Many testers come to the profession from non-technical backgrounds.
They may be subject matter experts who know their industry very well, but are not
specialists in software development. Without training or experience in programming or
system administration these individuals will likely struggle to make some open source
products work well for them.
When evaluating the skills needed to put open source software to good use look for the
following experience and training:
This list is not comprehensive, but it is typical of many open source products. In fact,
many of those languages and environments are open source projects themselves. There
are some open source tools that utilize C++, .NET, or VB languages, but they are not as
There may be developers in your company that have the skills that you need. However, it
is unlikely that your development group will have the manpower to spare for developers
to take on significant test tool implementation and management projects.
That leaves you with several possible outcomes from your skills evaluation. Ideally you
will have found people in your test group that have programming experience, and are not
daunted by the prospect of open source tools. Even if you don’t have people with the
necessary skills already, you may have people with the technical aptitude to be able to
learn to use the new tools quickly enough to make them worthwhile.
Unfortunately, there may be some QA teams that find that they just don’t have the skills
or time that are required by open source tools. It will still be valuable to include open
source products in your tool evaluations, but be aware of your limitations. That may
spare you from a failed implementation attempt and help limit your tool choices.
Picking the Right Tool
Now that you’ve figured out what kind of time you can spend on a tool implementation,
and what skills exist in your organization, you can begin the process of evaluating
products. Many of the tool evaluation procedures will be the same for open source and
for commercial tools, but there will be some differences in how you find the tools and
figure out whether they will work for you.
Define Your Requirements
QA groups have been telling their organizations for years how important it is for a
software project to have good requirements. Now it’s your turn to define what you need
a testing tool to do. These requirements will be based on your development environment,
languages, and the skills you have at your disposal.
It will be very helpful to define, in writing, what you require out of the software you
implement. Your requirements should cover:
• What you need to tool to do
• What products the tool should work with
• What measurements you want the tool to give you
• What skills the tool will require
Your list of requirements should be prioritized so that, in the very likely event that you
don’t find everything you want, you’ll know what you can live without.
Where to Find Open Source Test Tools
You’re not likely to find many advertisements for open source software in magazines,
and they’re probably not going to be exhibiting at tradeshows and conferences. But you
can find a lot of tools with a few simple searches on the Internet. What you may have
trouble finding are reliable reviews and comparisons.
The first place to start looking for open source testware is at http://opensourcetesting.org.
This site contains a fairly comprehensive catalog of open source testing software in a
number of different categories. There are brief descriptions of each of the more than 100
products, links to download sites, but no reviews. This site also contains some great links
to other web sites, some useful articles, and a forum where people discuss specific tools
as well as testing in general.
For more detailed information about free test tools you should investigate the Open
Testware Reviews site run by Danny Faught at http://tejasconsulting.com/open-testware/.
The reviews of test products there are very detailed, and there are excellent tool
comparisons and product category surveys available. Although this site does charge a
yearly subscription fee the cost is very reasonable (only $40), and there is some free
There are many other good sites with general information about testing tools, though they
may not specifically focus on open source products. There are large lists of tools at the
StickyMinds.com and TestingFAQs.org web sites. And a web search on your favorite
search engine for “open source software testing” will give you more than enough results
to investigate. Some of the best information you can get about tools will come from
people who have used them, so be sure to talk to people with experience.
What to Look For
Because of the large number of tools available you will not have time to do thorough
evaluations of every tool. You will want to research as much as you can to find out if a
product is right for you.
As with all test tool searches, the first thing you need to look for is whether the tool will
meet your defined requirements. You need to determine if it will support your
development languages and operating system platforms. You will also want to check
whether it has the features to accomplish what you need to the tool to do. Finally, you
will want to evaluate whether the product utilizes skills that you already have in your
organization. These are things you’ll want to look for regardless of the origin of the tool.
By comparing the tools you find to your defined requirements you avoid letting the
selection of tools drive your search.
The best way to know if a tool is going to work in your environment is to try it out.
Some open source tools are web based and have an online demo version that you can
experiment with. As your choices are narrowed down you will want to demonstrate the
appropriateness of the tool with a proof of concept.
With open source tools there are a few other things that you will need to consider in the
products you review. Danny Faught, author of the Open Testware Reviews, gives the
following suggestions for “How to spot a good free tool?
• Recently updated
• Good user community
• Decent web page home & documentation
• Reasonable installation process
• Bottom line: less cost to support than you'd pay a commercial tool vendor”6
This list underscores the risk that some open source products may not have continued
support. Many tools have strong beginnings but interest dies out and project
development is abandoned. This means that updates and fixes will not be available, and
makes the product more challenging to implement.
One open source web testing product, Enterprise Web Test, provides an excellent
example of this. At first glance this tool appears to have a lot of potential. Unlike most
open source web testing products that test a web site through direct HTTP calls to the
server, this tool claims to “‘drive’ the actual web browser on the actual platform they
intend to support.”7 However, further research reveals that the code for this product has
not been updated for more than 2 years. Although there are still people using Enterprise
Web Test, you need to be careful about the lack of continued development and support.
One last factor to consider in evaluating open source products is the license that is
associated with the software. Most people pay very little attention to user licenses, but
when it comes to open source tools the license may have a bigger impact than usual. This
is especially true if you plan to make modifications to the code. Open source licenses
stipulate how changes to the code must be managed, including sharing your changes with
the rest of the user community, and how the code can be used and distributed.
There are dozens of different open source licenses, including some standard ones like
GPL, LGPL, BSD, and MIT. The Open Source Initiative (http://opensource.org), or OSI,
is “a non-profit corporation dedicated to managing and promoting the Open Source
Definition for the good of the community.”8 Their web site provides a list of approved
open source licenses. A software product that is distributed with a license approved by
OSI can be considered as OSI Certified and can display a trademarked icon on the
distribution site. This is intended to protect users from software that claims to be “open
source” but doesn’t conform to standard definitions and licensing.
What’s Out There
The large number of open source software tools makes it difficult just to get a handle on
what is available. And new products are being developed all the time, complicating the
decision making process.
Considering that most open source products are created by developers to meet their own
needs, many open source test tools are for unit testing. And there are fewer tools for
testing Windows applications than products developed in Java. However, there are still a
good variety of very useful products in the most common, and most needed, software
testing tool categories.
There are a several good test management tools to help you organize test plans and test
cases, execute tests, and track results. They are usually web based applications that store
data in XML files or in a database. Some of these combine test management with an
automated test tool or defect tracking system.
FitNesse is one such product. Tests are developed through a wiki front end, a web
interface that allows people to create and edit web pages. Code is then written in Java or
C# to support the tests that are written.
A less complex open source test management tool is TestLink. This tool also works
through a web interface, and has a helpful tree structure for grouping tests. You can track
execution results, and there are simple metrics for reporting and summarizing test status.
Automated Functional/Regression Testing
The selection of functional or regression test automation tools is somewhat limited,
though it’s not due to a lack of tools in this category. The limitation is that most of them
are really unit test tools. They tend to exercise applications through back end
functionality, such as making direct calls to object methods, rather than through the user
For instance, most tools for testing web-based applications, such as Canoo WebTest or
WebInject, do so by sending HTTP calls to the server, rather than through the web
browser. This type of testing has its uses, but it doesn’t exercise the GUI of the
application, and validation of results is more difficult.
There are a few tools that do test an application’s user interface. Abbot and Jemmy are
open source tools for testing Java applications. Selenium is another product for testing
web applications in multiple browsers and platforms. It is still in the early stages of
development, but appears to have a lot of potential.
Several other test tools may look on the surface like they will provide good choices for
testing application interfaces, but most come with a hitch. For instance, QMTest is
backed by a strong development group and can do GUI testing, but to do so requires
extensive custom coding and it isn’t very straightforward to learn.
The performance testing category of tools is actually quite well supplied with good open
source products. Because of the high cost of commercial products, and developers’ own
interests in software performance, this class of tools has received a lot of attention from
the open source community.
JMeter, developed by the same group that created the Apache web server, is a Java
desktop application that can run on most operating systems. It provides excellent load
testing capabilities, with quite a bit of flexibility. It has a fairly intuitive interface, but
doesn’t provide much feedback during test execution.
Another very good open source performance test tool is OpenSTA (Open System Testing
Architecture). It has a small memory footprint for virtual users, and handles SSL
reasonably well. There are some good metrics built in and the ability to capture standard
Windows performance monitor statistics. However, the tool currently only runs on
Windows operating systems.
Both of these products have the ability to record test scripts, and tests can be distributed
across multiple machines. They provide very good choices for performance testing of
web systems. These are two of the best of many open source performance test tools.
There are also some very good choices of open source tools for defect tracking. Most of
these are web-based applications with very customizable interfaces. Bugzilla is
extremely popular, but Mantis is a good alternative. The GNATS set of tools provides
multiple user interfaces, including web and e-mail, but the server runs only on Unix.
There are so many good tools in this category that it is difficult at times to distinguish one
from the other.
In addition to these four main categories there are open source link checkers, tools for
security testing, and data generators. Checking product reviews and talking to people
who have made similar choices will help you make a more informed decision on a tool
from among the multitude of choices.
Costs of Ownership
Once you decide on an open source tool you still have a few challenges to overcome. As
has been already mentioned, the tools may be free but that doesn’t mean it won’t cost you
to use them. “While you will not need to pay a vendor to use the tools, they still have a
cost of ownership through evaluation, implementation, training and maintenance
Installation of many open source tools is not an easy, straightforward process. The
products are usually designed to run on multiple operating systems, and may involve
more than one component that will need to be manually installed and configured to work
correctly. For instance, many open source software systems require you to set up a web
server and database server, in addition to installing and configuring the product itself.
Once you have the product installed you have to figure out how to use it. You’re unlikely
to find training resources, although there are some companies that specialize in open
source product services. Fortunately, most decent open source products will have some
end user documentation to follow. You will probably expend considerable resources
experimenting with the software to get it to do what you want. And, because ease of use
is not always a top priority in open source projects, that experimentation is going to take
more time than with many commercial products.
Maintaining the open source product you’re using can also take more time than you may
be accustomed to. You are not usually required to register open source software, so you
may not receive notifications of updates. You may have to take the initiative to check for
updates, which may come more often than for commercial products. You will also have
to decide whether you want to take the risk of upgrading to the latest code base. This can
become more complicated if you have made modifications to the code to meet specific
needs. With more frequent product updates you can easily lose considerable time
keeping your open source software current. It’s not all bad news, though. The frequency
of updates can mean that you won’t have to wait as long for fixes and improvements that
you’ve been anticipating.
Open Source Software Risks
This paper has already covered a number of the benefits of using open source software as
well as a few of the risks. There are some additional hazards of open source products that
merit further discussion.
While many in the open source community tout the superiority of their products, there are
some products that are released without much attention to quality assurance practices. In
the haste to release new features and fixes, testing can be overlooked. And without a
company that can be held accountable for the quality of the software there may be little
feeling of responsibility. This is not usually the case, though, and we hope that this is
especially not the case with test tools.
Lack of support is another hazard of working with open source software. It is unlikely
that you can pick up the phone and talk to somebody about an issue you’re having.
Although open source developers can be very devoted and faithful to their products, the
development often happens in their spare time, so there are no guarantees about when and
if updates will be available.
Other potential risks with open source software include possibly poor usability, the
narrow focus of products, lack of documentation and training, and niches that may not be
filled. Not all open source products will have these kinds of problems, but they are some
things to pay attention to.
Despite the risks in using open source software, there is still tremendous potential in
these products. QA organizations can save a lot of money and still find powerful and
effective test tools, especially for defect tracking and performance testing. The
community involvement in developing these tools can create remarkable innovations and
high quality products.
Open source tools may require more time to implement, and more skilled people to use
them, than their commercial counterparts. However, if your company has the time and
skills needed, it is certainly worth the effort to consider them in your search. By carrying
out a careful and informed evaluation you can find some great products in the minefield
of open source testing tools.