Navigating the Minefield of Open Source Test Tools


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Navigating the Minefield of Open Source Test Tools

  1. 1. Navigating the Minefield of Open Source Test Tools By Jeff Jewell For STAREAST 2005 Introduction The process of testing software products can be greatly improved by the use of good tools. There are tools to help manage test plans and test cases, to automate regression and load testing, to track defects, and many other testing activities. Software quality can be increased, and the amount of time it takes to assure that quality can be reduced, by the implementation and proper use of software testing tools. However, selecting a software testing tool can often be an arduous process, but not usually because of a limited supply of available tools. The difficult part of selecting a tool is the dizzying number of available tools, and the wide variation in the costs and functionality of those tools. The real challenge becomes finding an affordable tool that can meet the complex needs of your development environment and products. The Testing Tools Landscape The landscape of software testing tools is dominated by a few high-end vendors with suites of tools that are very rich in features and relatively easy to use. These are usually the most well-known products, with advertising and sales forces pushing them into targeted organizations. Their high licensing costs (usually thousands of dollars for a single seat) put these tools out of reach for a large number of software development organizations. Struggling for light below the canopy of high-end tools are a number of lower cost products. These tools are typically not as feature laden as the high-end tools, but have much lower price tags. The mid-range tools tend to focus more narrowly on a specific market segment, such as automating only web applications and not Windows. Ease of use may also be an issue, but not necessarily. Those still not satisfied by what they find in the underbrush of mid-range test tools can emerge from the forest of commercial products to see a field populated with open source software. These tools appear to be a dream come true to cash-strapped development organizations. Most of them require no license fees, and even come with the source code so you can make your own enhancements. And there are dozens to choose from, touting advanced functionality and favorable comparisons to high-end products.
  2. 2. With visions of saving their companies thousands of dollars, software test teams rush headlong into a minefield of open source testing tools, only to discover some unexpected obstacles. The hidden costs and risks of using open source software can quickly erase the dreams they once had of a free software testing panacea. However, equipped with the right information, savvy test teams can find some very useful products they can afford. This paper will provide some of the information needed to successfully navigate the minefield of open source test tools. Some of the benefits and common misconceptions of open source software will be presented, along with information on what it takes to successfully use open source tools, how to find an appropriate tool, and what challenges may be faced in its implementation. What is Open Source Software? Open source software has publicly distributed source code, and is usually available for free. Having the source code freely available allows many developers to contribute to the development and enhancement of the product. Indeed, most open source products are developed by people who volunteer their time and efforts. Not all open source software is free, and not all free software is open source. There are a number of products that are distributed for free without the source code, and are usually known as “freeware”. Microsoft’s Internet Explorer is a good example of a freeware product. Microsoft does not publish the source code, but the software can be downloaded and used for free. “Shareware” products are also often available for free, but have some mechanism established to encourage or require users to eventually pay for the use of the product. On the other hand, you may be charged for some open source products, but the source code will be included. For more detailed definitions of open source software refer to the Open Source Definition at the Open Source Initiative.1 Open Source Acceptance The use of open source software is increasing, and a number of products have been accepted into the mainstream. Two of these very popular open source products have emerged as the poster children for the open source community. Apache web server now powers more than 60% of web sites2, clearly dominating a key market segment. The Linux operating system is also making significant gains in the marketplace, especially for systems where reliability is essential. Another open source product that is making waves in the software industry is Firefox, the latest web browser to be released based on the Mozilla code base. The 1.0 version of Firefox, which was released in November 2004, has been steadily gaining on the Microsoft Internet Explorer monopoly: “In the one-month period from Nov. 5, 2004 to Dec. 3, 2004, Firefox’s online usage share grew from 3.03 percent to 4.06 percent…. 1 2
  3. 3. “Firefox has made significant gains and Microsoft’s Internet Explorer’s usage share continues to fall in the United States. Between Nov. 5 and Dec. 3, Internet Explorer’s share dropped 1.09 percentage points to 91.80 percent, according to WebSideStory. This compares with a 0.32 percentage point drop the previous month. Since June 4, 2004, when IE’s usage share first began to drop due to security concerns, IE has lost a total of 3.68 percentage points.”3 Not only is Firefox usage growing, but corporate support for it is also increasing. Google recently hired one of the lead developers of Firefox, and committed 50% of his time to its continued development.4 The Lure of Open Source The biggest benefit of open source software is that it’s usually free. Although there are some costs associated with using open source tools (which will be discussed later), the fact that you don’t have to pay for licenses is a big draw for budget-conscious companies. Many people also like the fact that the source code for the software is openly published. Having access to the source code provides a number of advantages. First, it can give users a better understanding of how the software works internally. It also lets users customize the product, including adding new functionality. Finally, if there are problems with the software, users can fix work-stopping issues without having to wait for an update from other sources. Open source software products are distributed via the Internet, and are typically developed in small iterative cycles. Therefore, smaller and more frequent updates can be released than with most commercial products. Faster releases can mean that users won’t wait as long for enhancements and expected fixes. Finally, the collaborative nature of open source development creates a unique environment for software to evolve. With many people, from various backgrounds, contributing to the product development there is the opportunity for greater experimentation and innovation than with commercial products. In addition, most open source products are born from a real-world need and are developed by the people who plan to use them. This results in products that very closely match the needs of users, who may also find that they are able to get better response to suggestions for improvement. The Open Source Initiative web site offers the following explanation about the benefits of open source software: “The basic idea behind open source is very simple: When programmers can read, redistribute, and modify the source code for a piece of software, the software evolves. People improve it, people adapt it, people fix bugs. And this can happen at a speed that, if one is used to the slow pace of conventional software development, seems astonishing. 3 4
  4. 4. “We in the open source community have learned that this rapid evolutionary process produces better software than the traditional closed model, in which only a very few programmers can see the source and everybody else must blindly use an opaque block of bits.”5 How true this statement is really varies from one product to the next. But several features of open source development do have the potential to produce very good software. Many feel that the risks associated with open source software outweigh the benefits. There are a number of risks in open source products that will be discussed later. However, the perceived lack of quality in free products is not necessarily true of open source tools. The large community of people contributing to open source projects can help to overcome that hurdle. The fact that major corporations are willing to use open source software in mission critical applications is a great testimony to the quality and stability that these products can attain. Are You Ready for Open Source Software? After hearing all of the great reasons for adopting open source software products you may be anxious to get your hands on some. Before doing so, however, every company needs to evaluate if they are prepared to handle the investment it will take to be successful with open source tools. Depending on the people in your organization and the environment you work in, open source software may or may not be right for you. Have You Got the Time? Most major software acquisitions and implementations require an investment of time and money, and require certain skills. With commercial products the money spent on the product up front is often the biggest investment, and can shorten the amount of time and skills needed through training and professional services help. With open source software the investment of money will be less significant than the amount of time and the skills that will be required. Open source software products will frequently require more time to implement and maintain than commercial products because of a number of factors: • Installation and configuration processes may be unrefined • Training is not usually available • User documentation may be inadequate • Ease of use may not be a priority of the product • Frequent releases make it more difficult to stay up-to-date If your software team is pressed for time more than money then open source software may not be the best option for you. Have You Got the Skills? The other major factor to consider is whether you have the right skills in your organization to work with open source software. These products are usually created by 5
  5. 5. developers, for developers, and require more technical aptitude and expertise than their commercial counterparts. A large number of open source products have been developed by people with Unix and or Java backgrounds. As such, their products may present difficulties for people with only Windows and Microsoft language experience. Many have command line interfaces instead of a GUI, or require expertise in less-common programming languages. All of these factors combine to make products that may be very challenging for QA organizations. Many testers come to the profession from non-technical backgrounds. They may be subject matter experts who know their industry very well, but are not specialists in software development. Without training or experience in programming or system administration these individuals will likely struggle to make some open source products work well for them. When evaluating the skills needed to put open source software to good use look for the following experience and training: • XML • Perl • Python • Java • Jython • Unix • Linux This list is not comprehensive, but it is typical of many open source products. In fact, many of those languages and environments are open source projects themselves. There are some open source tools that utilize C++, .NET, or VB languages, but they are not as common. There may be developers in your company that have the skills that you need. However, it is unlikely that your development group will have the manpower to spare for developers to take on significant test tool implementation and management projects. That leaves you with several possible outcomes from your skills evaluation. Ideally you will have found people in your test group that have programming experience, and are not daunted by the prospect of open source tools. Even if you don’t have people with the necessary skills already, you may have people with the technical aptitude to be able to learn to use the new tools quickly enough to make them worthwhile. Unfortunately, there may be some QA teams that find that they just don’t have the skills or time that are required by open source tools. It will still be valuable to include open source products in your tool evaluations, but be aware of your limitations. That may spare you from a failed implementation attempt and help limit your tool choices.
  6. 6. Picking the Right Tool Now that you’ve figured out what kind of time you can spend on a tool implementation, and what skills exist in your organization, you can begin the process of evaluating products. Many of the tool evaluation procedures will be the same for open source and for commercial tools, but there will be some differences in how you find the tools and figure out whether they will work for you. Define Your Requirements QA groups have been telling their organizations for years how important it is for a software project to have good requirements. Now it’s your turn to define what you need a testing tool to do. These requirements will be based on your development environment, languages, and the skills you have at your disposal. It will be very helpful to define, in writing, what you require out of the software you implement. Your requirements should cover: • What you need to tool to do • What products the tool should work with • What measurements you want the tool to give you • What skills the tool will require Your list of requirements should be prioritized so that, in the very likely event that you don’t find everything you want, you’ll know what you can live without. Where to Find Open Source Test Tools You’re not likely to find many advertisements for open source software in magazines, and they’re probably not going to be exhibiting at tradeshows and conferences. But you can find a lot of tools with a few simple searches on the Internet. What you may have trouble finding are reliable reviews and comparisons. The first place to start looking for open source testware is at This site contains a fairly comprehensive catalog of open source testing software in a number of different categories. There are brief descriptions of each of the more than 100 products, links to download sites, but no reviews. This site also contains some great links to other web sites, some useful articles, and a forum where people discuss specific tools as well as testing in general. For more detailed information about free test tools you should investigate the Open Testware Reviews site run by Danny Faught at The reviews of test products there are very detailed, and there are excellent tool comparisons and product category surveys available. Although this site does charge a yearly subscription fee the cost is very reasonable (only $40), and there is some free content available. There are many other good sites with general information about testing tools, though they may not specifically focus on open source products. There are large lists of tools at the and web sites. And a web search on your favorite
  7. 7. search engine for “open source software testing” will give you more than enough results to investigate. Some of the best information you can get about tools will come from people who have used them, so be sure to talk to people with experience. What to Look For Because of the large number of tools available you will not have time to do thorough evaluations of every tool. You will want to research as much as you can to find out if a product is right for you. As with all test tool searches, the first thing you need to look for is whether the tool will meet your defined requirements. You need to determine if it will support your development languages and operating system platforms. You will also want to check whether it has the features to accomplish what you need to the tool to do. Finally, you will want to evaluate whether the product utilizes skills that you already have in your organization. These are things you’ll want to look for regardless of the origin of the tool. By comparing the tools you find to your defined requirements you avoid letting the selection of tools drive your search. The best way to know if a tool is going to work in your environment is to try it out. Some open source tools are web based and have an online demo version that you can experiment with. As your choices are narrowed down you will want to demonstrate the appropriateness of the tool with a proof of concept. With open source tools there are a few other things that you will need to consider in the products you review. Danny Faught, author of the Open Testware Reviews, gives the following suggestions for “How to spot a good free tool? • Recently updated • Good user community • Decent web page home & documentation • Reasonable installation process • Bottom line: less cost to support than you'd pay a commercial tool vendor”6 This list underscores the risk that some open source products may not have continued support. Many tools have strong beginnings but interest dies out and project development is abandoned. This means that updates and fixes will not be available, and makes the product more challenging to implement. One open source web testing product, Enterprise Web Test, provides an excellent example of this. At first glance this tool appears to have a lot of potential. Unlike most open source web testing products that test a web site through direct HTTP calls to the server, this tool claims to “‘drive’ the actual web browser on the actual platform they intend to support.”7 However, further research reveals that the code for this product has 6 7
  8. 8. not been updated for more than 2 years. Although there are still people using Enterprise Web Test, you need to be careful about the lack of continued development and support. One last factor to consider in evaluating open source products is the license that is associated with the software. Most people pay very little attention to user licenses, but when it comes to open source tools the license may have a bigger impact than usual. This is especially true if you plan to make modifications to the code. Open source licenses stipulate how changes to the code must be managed, including sharing your changes with the rest of the user community, and how the code can be used and distributed. There are dozens of different open source licenses, including some standard ones like GPL, LGPL, BSD, and MIT. The Open Source Initiative (, or OSI, is “a non-profit corporation dedicated to managing and promoting the Open Source Definition for the good of the community.”8 Their web site provides a list of approved open source licenses. A software product that is distributed with a license approved by OSI can be considered as OSI Certified and can display a trademarked icon on the distribution site. This is intended to protect users from software that claims to be “open source” but doesn’t conform to standard definitions and licensing. What’s Out There The large number of open source software tools makes it difficult just to get a handle on what is available. And new products are being developed all the time, complicating the decision making process. Considering that most open source products are created by developers to meet their own needs, many open source test tools are for unit testing. And there are fewer tools for testing Windows applications than products developed in Java. However, there are still a good variety of very useful products in the most common, and most needed, software testing tool categories. Test Management There are a several good test management tools to help you organize test plans and test cases, execute tests, and track results. They are usually web based applications that store data in XML files or in a database. Some of these combine test management with an automated test tool or defect tracking system. FitNesse is one such product. Tests are developed through a wiki front end, a web interface that allows people to create and edit web pages. Code is then written in Java or C# to support the tests that are written. A less complex open source test management tool is TestLink. This tool also works through a web interface, and has a helpful tree structure for grouping tests. You can track execution results, and there are simple metrics for reporting and summarizing test status. 8
  9. 9. Automated Functional/Regression Testing The selection of functional or regression test automation tools is somewhat limited, though it’s not due to a lack of tools in this category. The limitation is that most of them are really unit test tools. They tend to exercise applications through back end functionality, such as making direct calls to object methods, rather than through the user interface. For instance, most tools for testing web-based applications, such as Canoo WebTest or WebInject, do so by sending HTTP calls to the server, rather than through the web browser. This type of testing has its uses, but it doesn’t exercise the GUI of the application, and validation of results is more difficult. There are a few tools that do test an application’s user interface. Abbot and Jemmy are open source tools for testing Java applications. Selenium is another product for testing web applications in multiple browsers and platforms. It is still in the early stages of development, but appears to have a lot of potential. Several other test tools may look on the surface like they will provide good choices for testing application interfaces, but most come with a hitch. For instance, QMTest is backed by a strong development group and can do GUI testing, but to do so requires extensive custom coding and it isn’t very straightforward to learn. Performance Testing The performance testing category of tools is actually quite well supplied with good open source products. Because of the high cost of commercial products, and developers’ own interests in software performance, this class of tools has received a lot of attention from the open source community. JMeter, developed by the same group that created the Apache web server, is a Java desktop application that can run on most operating systems. It provides excellent load testing capabilities, with quite a bit of flexibility. It has a fairly intuitive interface, but doesn’t provide much feedback during test execution. Another very good open source performance test tool is OpenSTA (Open System Testing Architecture). It has a small memory footprint for virtual users, and handles SSL reasonably well. There are some good metrics built in and the ability to capture standard Windows performance monitor statistics. However, the tool currently only runs on Windows operating systems. Both of these products have the ability to record test scripts, and tests can be distributed across multiple machines. They provide very good choices for performance testing of web systems. These are two of the best of many open source performance test tools. Defect Tracking There are also some very good choices of open source tools for defect tracking. Most of these are web-based applications with very customizable interfaces. Bugzilla is
  10. 10. extremely popular, but Mantis is a good alternative. The GNATS set of tools provides multiple user interfaces, including web and e-mail, but the server runs only on Unix. There are so many good tools in this category that it is difficult at times to distinguish one from the other. In addition to these four main categories there are open source link checkers, tools for security testing, and data generators. Checking product reviews and talking to people who have made similar choices will help you make a more informed decision on a tool from among the multitude of choices. Costs of Ownership Once you decide on an open source tool you still have a few challenges to overcome. As has been already mentioned, the tools may be free but that doesn’t mean it won’t cost you to use them. “While you will not need to pay a vendor to use the tools, they still have a cost of ownership through evaluation, implementation, training and maintenance costs….”9 Installation of many open source tools is not an easy, straightforward process. The products are usually designed to run on multiple operating systems, and may involve more than one component that will need to be manually installed and configured to work correctly. For instance, many open source software systems require you to set up a web server and database server, in addition to installing and configuring the product itself. Once you have the product installed you have to figure out how to use it. You’re unlikely to find training resources, although there are some companies that specialize in open source product services. Fortunately, most decent open source products will have some end user documentation to follow. You will probably expend considerable resources experimenting with the software to get it to do what you want. And, because ease of use is not always a top priority in open source projects, that experimentation is going to take more time than with many commercial products. Maintaining the open source product you’re using can also take more time than you may be accustomed to. You are not usually required to register open source software, so you may not receive notifications of updates. You may have to take the initiative to check for updates, which may come more often than for commercial products. You will also have to decide whether you want to take the risk of upgrading to the latest code base. This can become more complicated if you have made modifications to the code to meet specific needs. With more frequent product updates you can easily lose considerable time keeping your open source software current. It’s not all bad news, though. The frequency of updates can mean that you won’t have to wait as long for fixes and improvements that you’ve been anticipating. 9
  11. 11. Open Source Software Risks This paper has already covered a number of the benefits of using open source software as well as a few of the risks. There are some additional hazards of open source products that merit further discussion. While many in the open source community tout the superiority of their products, there are some products that are released without much attention to quality assurance practices. In the haste to release new features and fixes, testing can be overlooked. And without a company that can be held accountable for the quality of the software there may be little feeling of responsibility. This is not usually the case, though, and we hope that this is especially not the case with test tools. Lack of support is another hazard of working with open source software. It is unlikely that you can pick up the phone and talk to somebody about an issue you’re having. Although open source developers can be very devoted and faithful to their products, the development often happens in their spare time, so there are no guarantees about when and if updates will be available. Other potential risks with open source software include possibly poor usability, the narrow focus of products, lack of documentation and training, and niches that may not be filled. Not all open source products will have these kinds of problems, but they are some things to pay attention to. Conclusion Despite the risks in using open source software, there is still tremendous potential in these products. QA organizations can save a lot of money and still find powerful and effective test tools, especially for defect tracking and performance testing. The community involvement in developing these tools can create remarkable innovations and high quality products. Open source tools may require more time to implement, and more skilled people to use them, than their commercial counterparts. However, if your company has the time and skills needed, it is certainly worth the effort to consider them in your search. By carrying out a careful and informed evaluation you can find some great products in the minefield of open source testing tools.