Open Source Software Testing Tools
Upcoming SlideShare
Loading in...5
×
 

Open Source Software Testing Tools

on

  • 1,319 views

Software Test automation tools are available under several categories such as commercial, free software, open source software and etc. In this paper Open Source Software Testing Tools will be ...

Software Test automation tools are available under several categories such as commercial, free software, open source software and etc. In this paper Open Source Software Testing Tools will be discussed.

Open source software test automation tools may be practical alternatives to popular closed-source commercial applications and some open source tools offers features or performance benefits that exceed their commercial counterparts. The source code is openly published for use and/or modification from its original design, free of charge. And these are usually available under a license defined by the Open Source Initiative.

Statistics

Views

Total Views
1,319
Views on SlideShare
1,319
Embed Views
0

Actions

Likes
1
Downloads
12
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Open Source Software Testing Tools Open Source Software Testing Tools Document Transcript

  • ICT 3003 – Software Quality Assurance Assignment I Open Source Software Quality Assurance Testing Tools University of Colombo School of Computing Prepared by Waruna Kodituwakku
  • Page | 1 Table of Contents 1. Introduction............................................................................................................................2 1.1 Open Source Software Testing Tools ..............................................................................2 1.2 Types of Software Testing Tools.....................................................................................2 2. Popular Open Source Software Testing Tools.......................................................................3 2.1 Selenium ......................................................................................................................3 2.2 FitNesse........................................................................................................................3 2.3 Watir ............................................................................................................................4 2.4 Apache JMeter .............................................................................................................4 2.5 Robotium......................................................................................................................4 2.6 Bugzilla........................................................................................................................5 2. Comparison of Selenium, Bugzilla and JMeter .....................................................................5 2.1 Analysis of Strengths and Weaknesses of Selected Tools...............................................6 2.1.1 Selenium ...................................................................................................................6 2.1.2 Bugzilla.....................................................................................................................6 2.1.3 JMeter .......................................................................................................................7 3. Future of Software Testing Tools ..........................................................................................7 4. Conclusion .............................................................................................................................7 5. References..............................................................................................................................8
  • Page | 2 1. Introduction Software Quality Assurance (SQA) involves monitoring and improving the software testing process, making sure that any agreed-upon standards and procedures are followed, and ensuring that problems are found and treated. In today’s context SQA is essential to every software development process. Software Testing is a costly and time consuming process in software development. Therefore, software testing tools are often deployed to automate the process in order to reduce cost and improve efficiency. SQA now incorporates more sophisticated tools that enhance software testing capabilities. Testing automation tools enable and help developers and testers to automate the entire process of testing in software development which include; find and report defects and issues faster; confirm tests meet customer specifications and requirement-traceability; have a fine way for communication; archive the test plan and execute it manually or automatically; maintain reusable tests; and document, verify and audit which tests were run for each project. 1.1 Open Source Software Testing Tools Software Test automation tools are available under several categories such as commercial, free software, open source software and etc. In this paper Open Source Software Testing Tools will be discussed. Open source software test automation tools may be practical alternatives to popular closed- source commercial applications and some open source tools offers features or performance benefits that exceed their commercial counterparts. The source code is openly published for use and/or modification from its original design, free of charge. And these are usually available under a license defined by the Open Source Initiative. 1.2 Types of Software Testing Tools Software testing tools can be categorized under several bases. Following table is shows software testing tools categorized under the testing activity or the process in which they are used. Type Description Open Source Examples Functional Testing Functional testing is the process of confirming the functionality of the application. This type of testing ignores the internal parts and focuses on the output and determines that it is as per requirement or not. Selenium, Watir, WatiN, Canoo WebTest, Cucumber Source Code Testing Source Code Testing tools are designed to analyze source code and/or compiled version to verify the source code and help identify the security flaws. SWAAT, Flawfinder, PMD, Brakeman, RIPS, RATS Performance Testing To check whether system meets performance requirements Apache JMeter, FunkLoad, OpenSTA Database Testing Database testing is about checking exact values which have been retrieved from the database by the web or desktop application. Data should exactly match with those in the records that are stored in the Database Database Opensource Test Suite, DBMonster, DB Sanity, DBFeeder Bug Tracking Tools These software applications are designed to help keep track of reported software bugs in software development efforts. Bugzilla, Mantis, Trac, Redmine, Request Tracker,
  • Page | 3 Test Management Tools Test management tools are used to structure automated tests and manual test processes, and to manage multiple environments easily. Quality assurance teams use these types of tools as a single application for managing test cases, environments, automated tests, defects and project tasks. Test Manager, TETware, Requirements and Testing Hub (RTH) 2. Popular Open Source Software Testing Tools 2.1 Selenium  Category : Functional Test Tool, Web Application Testing  License: Apache License 2.0 Selenium is composed of multiple software tools. Those are Selenium IDE, Selenium WebDirver and Selenium Remote Control. Selenium is a portable software testing framework for web applications.. It supports a number of accepted programming languages, including C#, Java, Groovy, Perl, PHP, Python and Ruby for writing test cases. The tests can then be run against many modern web browsers. Selenium is deployed on Windows, Linux, and Macintosh platforms. Features  Record and playback  Intelligent field selection  Auto complete for all common Selenium commands  Walk through tests, debug and set breakpoints  Save tests as Selenese, Ruby scripts, or other formats  Option to automatically assert the title of every page  Easy customization through plug-ins 2.2 FitNesse  Category : Test Management Tools  License : Common Public License (CPL) This tool allows users to specify and run acceptance tests for software systems (an agile style of black-box testing acceptance and regression testing). FitNesse facilitates detailed readable description of system function and it supports programming languages such as Java, C++, Python, Ruby, Delphi, C#, etc. Features  Wiki engine  Embedded web server  Test engines (FIT, SLIM)  Automated input generation and test case creation  Additional libraries
  • Page | 4 2.3 Watir  Category: Functional Test Tool, Web Application Testing  License: BSD License Watir is a free, open-source functional testing tool for automating browser-based tests of web applications. It works with applications written in any language. Watir project consists of several smaller projects. Some of them are Watir-classic, Watir-webdriver and Watirspec. Features  Handles multiple browser windows with attach method.  Supports JavaScript events  Supports frames and nested frames  Supports visible and invisible running of test cases  Automatic page loading waiting  Supports hidden fields and saving images in a page and Screen capturing 2.4 Apache JMeter  Categories: Performance Testing, Load Testing  License: Apache License 2.0 Apache JMeter is a 100% pure Java desktop application designed to load test functional behavior and measure performance. It was originally designed for testing Web Applications but has since expanded to other test functions. JMeter architecture is based on plugins. Most of its features are implemented with plugins. Off-site developers can easily extend JMeter with custom plugins. Features  Can load and performance test many different server types (Web - HTTP, HTTPS, SOAP, Database via JDBC, LDAP, JMS, Mail - SMTP(S), POP3(S) and IMAP(S), Native commands or shell scripts)  Complete portability and 100% Java purity.  Full multithreading framework allows concurrent sampling by many threads and simultaneous sampling of different functions by separate thread groups. Careful GUI design allows faster operation and more precise timings.  Caching and offline analysis/replaying of test results.  Highly Extensible 2.5 Robotium  Categories : Mobile Testing tool, Android Application Test Tool  License: Apache License 2.0 Robotium is a test framework created to make it easy to write powerful and robust automatic black-box test cases for Android applications. With the support of Robotium, test case developers can write function, system and acceptance test scenarios, spanning multiple Android activities.
  • Page | 5 Features  Fast test case execution.  The framework handles multiple Android activities automatically.  Minimal time needed to write solid test cases.  Readability of test cases is greatly improved, compared to standard instrumentation tests.  Test cases are more robust due to the run-time binding to GUI components.  Integrates smoothly with Maven or Ant to run tests as part of continuous integration 2.6 Bugzilla  Category: Bug Tracking Tool  License: Mozilla Public License Bugzilla is a web-based bug tracker and testing tool. This allows individuals or groups of developers to keep track of outstanding bugs in their product effectively. Generally Bugzilla tracks bugs and code changes, communicates with teammates, submits and reviews patches and manages quality assurance. Bugzilla is written in Perl, and works on various databases including MySQL and Oracle. Features  Advanced search capabilities  Email notifications controlled by user preferences and integrated email capabilities  Bug lists in multiple formats (Atom, iCal, etc.) and reports and charts  Automatic Duplicate Bug Detection  Time Tracking, Request System, Private Attachments and Comments  Automatic Username Completion or Drop-Down User Lists and Patch Viewer  Multiple Authentication Methods and Support for Multiple Database Engines  2. Comparison of Selenium, Bugzilla and JMeter Selenium (IDE, WebDriver and Remote Control) BugZilla Apache JMeter Testing Type Functional Testing, Regression Testing Bug Tracking Performance Testing, Load Testing Developed by ThoughtWorks Inc. Mozilla Foundation Apache Software Foundation Written in Java (Programming Language) Perl (Programming Language) Java (Programming Language) Operating System Cross-platform (Windows, Linux, OS X, Solaris) Cross-platform Cross-platform User Interface Available Integrated into ALM & IDE, Standalone application,Web based interface Web based Interface Batch Mode, Stand Alone application Record and Playback Option Available No Available
  • Page | 6 Application Type It can be used to test only web based applications. It can be used to test only web based applications It can test web-based as well as desktop applications. Integration with other tools Allowed Allowed Allowed Report Generation Facility Available Available Available Multilingual (Localization) No Yes No Plugins, Add-ons, Customizations Available Available Available Installation Required Installation Required Installation Required Installation Required Documentation and Community Support Available Available Available 2.1 Analysis of Strengths and Weaknesses of Selected Tools 2.1.1 Selenium Strengths Selenium is one of the most popular open-source automation tools. Selenium provides a handy unified interface that works with a large number of browsers, and allows writing tests in almost all the languages (E.g. Java, C#, Python, JavaScript, PHP, Erlang and etc). Weaknesses Selenium is written in JavaScript which causes a significant weakness: browsers impose a pretty strict security model on any JavaScript that they execute in order to protect a user from malicious scripts. And also it doesn't work directly with Flash. 2.1.2 Bugzilla Strengths Main strength of Bugzilla is that it is one of the well established and widely used software in the industry. Other key strengths of BugZilla can be identified as; easily understood, easily integrates with several test management tools (e.g. Test Link, Test Runner, Testopia), very test friendly and contains almost all the features found in commercial products. Weaknesses Main weakness of Bugzilla is its complex installation. And even with improvements; Bugzilla user interface hasn't changed much over the years – a fact that is not attractive for end-users.
  • Page | 7 2.1.3 JMeter Strengths JMeter is an Open Source Tool and it also a Freeware. It supports the distributed testing and has various target systems and further it is Cross-platform. It has very high extensibility too, because pluggable samplers allow unlimited testing capabilities. Overall JMeter is a stable, easy to comprehend and long standing desktop Java application. Weaknesses It’s always advisable not to run JMeter on the same machine running the application to be tested. JMeter may use extensive resources that might affect the other application's performance. JMeter chart representation looks a bit confusing to some users and terminology appears to be not very clear. Getting "correct" response times from JMeter process is very complex. Also remote machines must be declared in a property file before starting application. 3. Future of Software Testing Tools Today, software has changed from a standard, single, windows-based platform to multiple platforms. When considering about growth of software test automation tools we can see that there will be new areas which will enhance user friendliness, integration between tools, combining different tools to fit user needs. With script-less automation, new tools will be shaped so that users could setup automation testing in a simple and logical way. It means that technical skills and programming knowledge will not be required any more. Another aspect is that Visual Test Automation is the future for test object recognition in the modern software application development and testing world. It contains algorithms built on OCR (object character recognition), ICR (image character recognition) and gesture recognition technology (interpreting human gestures). Technology based on visual based object recognition will allow testers to execute scripts cross-client and cross-platform. Automated testing tools will become less complex, easy to use and adaptive. Users will not need weeks to learn how to use testing tools. This kind of software testing will be even faster and will give exact results. 4. Conclusion To be successful in the challenging business of software development, it is essential to provide a quality product to the customers. Such process is known as ‘Software Quality Assurance’ (SQA). Software Testing Tools play a vital role in SQA. Testing is more than just debugging. Testing is used to locate defects and correct them and also in validation, verification process, and reliability measurement. There are commercial and free testing tools as well as open source testing tools. Among them, Open Source Testing Tools seems to the most popular. Comparatively it provides more flexibility, more eases of usage and allows tool customization also. In this paper several popular and notable open source software testing tools have been categorized, analyzed, compared and their respective strengths and weaknesses discussed. The future of the testing tools is indicative that they will become less complex, more easy to use more adaptive and faster too.
  • Page | 8 5. References 1. Open Source Testing. 2013. Open source functional testing tools. [ONLINE] Available at: http://www.opensourcetesting.org/functional.php. [Accessed 15 March 2013]. 2. VTB. 2013. Article - Test Automation Tools. [ONLINE] Available at: http://www.vietnamesetestingboard.org/zbxe/?document_srl=555503. [Accessed 25 March 2013]. 3. Wikipedia, the free encyclopedia. 2013. Test automation - Wikipedia, the free encyclopedia. [ONLINE] Available at: http://en.wikipedia.org/wiki/Test_automation. [Accessed 25 March 2013]. 4. SparkWiz. 2013. The Types and Benefits of Software Testing Tools - SparkWiz | Spark Wiz. [ONLINE] Available at: http://sparkwiz.com/tips-n-tricks/the-types-and- benefits-of-software-testing-tools/. [Accessed 25 March 2013]. 5. OWASP. 2013. Source Code Analysis Tools - OWASP. [ONLINE] Available at: https://www.owasp.org/index.php/Source_Code_Analysis_Tools. [Accessed 25 March 2013]. 6. Google Project Hosting. 2013. robotium - The world's leading Android™ test automation framework - Google Project Hosting . [ONLINE] Available at: http://code.google.com/p/robotium/. [Accessed 25 March 2013]. 7. Wikipedia, the free encyclopedia. 2013. Bugzilla - Wikipedia, the free encyclopedia. [ONLINE] Available at: http://en.wikipedia.org/wiki/Bugzilla. [Accessed 25 March 2013]. 8. TheServerSide.com. 2013. Open source software testing tools news, help and research - TheServerSide.com. [ONLINE] Available at: http://www.theserverside.com/resources/Open-source-software-testing-tools. [Accessed 25 March 2013]. 9. IndianGNU. 2013. IndianGNU.org » Blog Archive » Top 5 Open Source Software Testing QA tools. [ONLINE] Available at: http://www.indiangnu.org/2011/top-5-open- source-software-testing-qa-tools/. [Accessed 25 March 2013]. 10. Xoriant Software Product Engineering. 2013. Selenium Open Source Test Automation Tool: An Overview | Xoriant Software Product Engineering. [ONLINE] Available at: http://www.xoriant.com/blog/software-testing-and-qa/selenium-open-source-test- automation-tool-an-overview.html. [Accessed 25 March 2013]. 11. Apache JMeter™. 2013. Apache JMeter - Apache JMeter™. [ONLINE] Available at: http://jmeter.apache.org/index.html. [Accessed 25 March 2013].
  • Page | 9 12. CSS Corp Labs. 2013. CSS Corp Labs: Testing Tools Comparison. [ONLINE] Available at: http://cssinnovations.blogspot.com/2011/10/testing-tools- comparison_27.html. [Accessed 25 March 2013]. 13. ProgrammingZ.com. 2013. Open-Source Quality Assurance And Performance Analysis Tools . [ONLINE] Available at: http://www.programmingz.com/b/2435541/Open-Source-Quality-Assurance-And- Performance-Analysis-Tools/#.UU-rkaAtg3I. [Accessed 25 March 2013]. 14. Open Source Testing. 2013. Open source test management tools. [ONLINE] Available at: http://www.opensourcetesting.org/testmgt.php. [Accessed 25 March 2013]. 15. PHP Unit Manual. 2013. Chapter 8. Database Testing . [ONLINE] Available at: http://www.phpunit.de/manual/3.6/en/database.html. [Accessed 25 March 2013]. 16. Mobile QA Zone. 2013. Visual Test Automation Leads the Future of the Software Testing Industry - Mobile QA Zone. [ONLINE] Available at: http://www.mobileqazone.com/profiles/blogs/visual-test-automation-leads-the-future-of- the-software-testing. [Accessed 25 March 2013]. 17. Software QA and Testing Resource Center. 2013. Database Testing Tools. [ONLINE] Available at: http://sqa.fyicenter.com/FAQ/Testing-Tools/Database_Testing_Tools.html. [Accessed 25 March 2013]. 18. Ryan Schuetzler. 2013. QA Tool Review: JMeter. [ONLINE] Available at: http://www.schuetzler.net/2010/01/02/jmeter/. [Accessed 25 March 2013]. 19. Wikipedia, the free encyclopedia. 2013. Comparison of issue-tracking systems - Wikipedia, the free encyclopedia. [ONLINE] Available at: http://en.wikipedia.org/wiki/Comparison_of_issue-tracking_systems. [Accessed 25 March 2013]. 20. Jon's Technical Notes. 2013. Jon's Technical Notes: Pros and Cons of Selenium. [ONLINE] Available at: http://jdrew33.blogspot.com/2012/02/pros-and-cons-of- selenium.html. [Accessed 25 March 2013].