Performance Test WCF/WPF app - Selecting right Tool

5,293 views

Published on

Pick the right tool for Performance Testing a WCF and WPF based .net application. This paper discusses 107 load testing tools and frameworks to performance test a SOA based, WCF/WPF/Silverlight application.

Published in: Technology
  • Be the first to comment

Performance Test WCF/WPF app - Selecting right Tool

  1. 1. Software Quality Assurance Plan For WCF/WPF based Application Version 0.1 May 15, 2014 Compiled by: Kamran Shaukat Ali Khan Chrome Technologies ChromeIS.com
  2. 2. Software Quality Assurance Plan Performance Test Plan – Ascent –Version 0.1 Table of Contents Document Information................................................................................................vi Document Information.................................................................................................vi Document Review Information.....................................................................................vi Document Review Information.....................................................................................vi Document Revision History..........................................................................................vi Document Revision History...........................................................................................vi List of Figures..........................................................................................................vii List of Figures.............................................................................................................vii List of Tables............................................................................................................vii List of Tables...............................................................................................................vii 1. Abstract.....................................................................................................................1 1.1 Proposed Solution...................................................................................................1 2. Performance Testing Tools Evaluated.........................................................................2 2.1 HP LoadRunner and Performance Center....................................................................2 2.2 IBM Rational Performance Tester..............................................................................3 2.3 Borland Silk Performer............................................................................................3 2.4 Telerik Test Studio..................................................................................................3 2.5 Apache JMeter.......................................................................................................3 2.6 j-Chav..................................................................................................................4 2.7 Microsoft WCAT – Web Capacity Analysis Tool (formerly WAS – Web Application Stress Tool)..........................................................................................................................4 2.8 Sun VisualVM.........................................................................................................4 2.9 WAPT...................................................................................................................5 2.10 Web Application Load Simulator (LoadSim)...............................................................5 2.11 Neo Load.............................................................................................................5 2.12 EggPlant Load Tester.............................................................................................5 2.13 OpenSTA – Open System Testing Architecture .........................................................6 2.14 OpenWebLoad......................................................................................................6 2.15 OpenLoad............................................................................................................6 2.16 loadUI.................................................................................................................6 2.17 LoadeaTest..........................................................................................................6 2.18 LoadManager.......................................................................................................7 2.19 Test Automation FX...............................................................................................7 2.20 CodedUI..............................................................................................................7 2.21 Soap UI...............................................................................................................7 2.22 Soap-Stone.........................................................................................................7 2.23 Web Performance Suite.........................................................................................8 2.24 Webserver Stress Tool...........................................................................................8 2.25 WebLOAD............................................................................................................8 2.26 QALoad...............................................................................................................8 2.27 Lisa Load.............................................................................................................8 2.28 Grinder...............................................................................................................9 2.29 nGrinder..............................................................................................................9 Copyrights © NetSol Technologies Ltd. All rights reserved iii Unauthorized copy or use of this document is strictly prohibited
  3. 3. Software Quality Assurance Plan Performance Test Plan – Ascent –Version 0.1 2.30 GrinderStone.......................................................................................................9 2.31 MStone...............................................................................................................9 2.32 ValGrind..............................................................................................................9 2.33 QEngine..............................................................................................................9 2.34 Qtest................................................................................................................10 2.35 HammerHead 2..................................................................................................10 2.36 Hammerora........................................................................................................10 2.37 Pylot.................................................................................................................10 2.38 OpenWebLoad....................................................................................................10 2.39 SD’s Profiler.......................................................................................................11 2.40 Benerator..........................................................................................................11 2.41 SOATest (by Parasoft).........................................................................................11 2.42 Scapa Test and Performance Platform ...................................................................11 2.43 CLIF Load Injection ............................................................................................12 2.44 ContiPerf...........................................................................................................12 2.45 Coveriety...........................................................................................................12 2.46 D-ITG (Distributed Internet Traffic Generator).........................................................12 2.47 Database Opensource Test Suit (DOTS).................................................................13 2.48 DB Monster........................................................................................................13 2.49 Deluge..............................................................................................................13 2.50 DieselTest..........................................................................................................13 2.51 Faban................................................................................................................13 2.52 FunkLoad...........................................................................................................14 2.53 Curl-Loader........................................................................................................14 2.54 Spirant Avalanche...............................................................................................14 2.55 IXIA IxLoad........................................................................................................14 2.56 Paessler.............................................................................................................14 2.57 AQ TestComplete and AQ Time.............................................................................14 2.58 Test Maker.........................................................................................................15 2.59 TP Test..............................................................................................................15 2.60 Tsung................................................................................................................15 2.61 Tnsgen..............................................................................................................15 2.62 N-Time..............................................................................................................15 2.63 p-unit................................................................................................................16 2.64 Lobo.................................................................................................................16 2.65 MessAdmin........................................................................................................16 2.66 Multi Router Traffic Grapher (MRTR)......................................................................16 2.67 RRRD Tool.........................................................................................................16 2.68 Multi-Mechanize..................................................................................................16 2.69 Stress Driver......................................................................................................17 2.70 Load Impact.......................................................................................................17 2.71 Pure Load..........................................................................................................17 2.72 Pure Agent.........................................................................................................17 2.73 FWPTT...............................................................................................................17 2.74 JBlits Pro...........................................................................................................18 2.75 JCrawler............................................................................................................18 Copyrights © NetSol Technologies Ltd. All rights reserved iv Unauthorized copy or use of this document is strictly prohibited
  4. 4. Software Quality Assurance Plan Performance Test Plan – Ascent –Version 0.1 2.76 PandoraFMS.......................................................................................................18 2.77 Siege................................................................................................................18 2.78 HTTP Load.........................................................................................................18 2.79 httPerf...............................................................................................................19 2.80 iPerf..................................................................................................................19 2.81 Site Blaster........................................................................................................19 2.82 RawLoadTester...................................................................................................19 2.83 Allmon..............................................................................................................20 2.84 AppsWatch.........................................................................................................20 2.85 BurnIn Test........................................................................................................20 2.86 Capacity Calibration (CapCal)...............................................................................20 2.87 Chariot..............................................................................................................21 2.88 CitraTestVU........................................................................................................21 2.89 SSW Performance Pro..........................................................................................21 2.90 e-Load..............................................................................................................21 2.91 MLoad...............................................................................................................21 2.92 Monitor Master...................................................................................................22 2.93 Forecast............................................................................................................22 2.94 GS DataGenerator...............................................................................................22 2.95 vPerformer.........................................................................................................22 2.96 JUnit.................................................................................................................22 2.97 Xceptance LoadTest............................................................................................23 2.98 zapwireless........................................................................................................23 2.99 Seagull..............................................................................................................23 2.100 SIPp................................................................................................................23 2.101 SLAMD............................................................................................................23 2.102 IxoraRMS.........................................................................................................24 2.103 j-hawk.............................................................................................................24 2.104 Wireshark........................................................................................................24 2.105 Ostinato...........................................................................................................24 2.106 Eclipse Test & Performance Tools Platform Project (EPTP)........................................24 2.107 Web Polygraph.................................................................................................24 3. Tools Selection Criteria.............................................................................................25 3.1 Business and Technical Requirements......................................................................25 3.2 Performance Test Objectives..................................................................................25 3.3 Performance Test Execution...................................................................................28 3.4 Load Description...................................................................................................28 4. Conclusion................................................................................................................29 4.1 Selected Tools......................................................................................................30 4.2 SWOT Analysis.....................................................................................................30 4.3 Training Needs.....................................................................................................31 4.4 Lessons Learnt.....................................................................................................31 5. Definition of Terms, Acronyms And Abbreviations.....................................................31 Copyrights © NetSol Technologies Ltd. All rights reserved v Unauthorized copy or use of this document is strictly prohibited
  5. 5. Software Quality Assurance Plan Performance Test Plan – Ascent –Version 0.1 Document & Data Control Document InformationDocument Information Category Information <Title> Performance Test Tool Selection <Version> 0.1 <Author> Kamran Shaukat Ali Khan <Identifier> <Life Cycle Activity> TLC <Issue Date> Pending Document Review InformationDocument Review Information Review Date Reviewer Name Version Reference / Evidence Document Revision HistoryDocument Revision History Author Date Version Description Kamran Shaukat Ali Khan 15 May 2014 0.1 Draft Copyrights © NetSol Technologies Ltd. All rights reserved vi Unauthorized copy or use of this document is strictly prohibited
  6. 6. Software Quality Assurance Plan Performance Test Plan – Ascent –Version 0.1 List of FiguresList of Figures Figure 1, Performance Acceptance at a glance..............................................................27 List of TablesList of Tables Table 1. Tools fostering Performance Testing phases.....................................................2 Table 2. Selected tools for Performance Testing phases................................................30 Copyrights © NetSol Technologies Ltd. All rights reserved vii Unauthorized copy or use of this document is strictly prohibited
  7. 7. Software Quality Assurance Plan Performance Testing – Tool Selection 1. Abstract OUR PRODUCT is engineered for performance and agility. Performance testing is the process of assessing performance/efficiency of a transaction system whilst simultaneous users are accessing it; this is to determine if subject application (with all factors involved) can serve its purpose efficiently. Performance Testing is a fundamental aspect of software engineering, but it is a practice that too often is easily forgotten in today’s fast-paced application development culture. Performance testing applications efficiently is essentially an integral component for meeting end customer expectations. The aim of this document is to evaluate and compare all available performance testing tools to determine their usability and effectiveness. Software testing is a crucial part of software development and the process of automating software testing is vital to its success. OUR PRODUCT is a complex system and performance testing throughout the software development cycle is pertinent to the success of the software. To enable comparison of the automated performance testing tools, a multi-partitioned metric suite was developed to guide the tool evaluation process. Inferences, implications and results are presented and discussed in Section 3. 1.1 Proposed Solution The overall goal of this research is to evaluate and compare several automated performance testing tools to determine their usability and effectiveness for OUR PRODUCT. To accomplish this goal, the research: 1. Prepared a comprehensive performance testing tools available on internet. 2. Broadened the scope of tools to include shareware, freeware, Open licenses and open source tools. The list of tools, along with tools’ assessment is described in Section 2. 3. Defined criteria for tools selection based on business and technical requirements of OUR PRODUCT; this will help narrow down the scope of research. 4. Compared existing tools in practice at ChromeIS with evolving requirements as highlighted in above point. 5. Performed assessment of tools based on features and later, experiment with actual application i.e. OUR PRODUCT (for e.g. CMS) 6. Tested the target applications using the selected automated testing tools and gather resulting data 7. Evaluated and interpret results to draw inferences and make recommendations OUR PRODUCT is a smart client being developed in Visual Studio 2010 using.net framework 4.5 – this imposes certain limitations. An RND activity has been concluded where a number of tools were evaluated. The assessment concluded that Performance Testing has to be carried out in Visual Studio .net itself. However, there are certain other tools which can be utilized to foster in Performance Testing. Below table enlists tools which will be used for Performance Testing as well as for resources monitoring: Copyrights © NetSol Technologies Ltd. All rights reserved 1 Unauthorized copy or use of this document is strictly prohibited
  8. 8. Software Quality Assurance Plan Performance Testing – Tool Selection Tool Purpose Visual Studio 2013 For Performance Testing of areas which can’t be worked out with HP PC / LR. HP LoadRunner 12 Automated performance and stress testing NeoLoad Trace client/server communication SolarWind Defects logging and tracking Spotlight Enterprise Resource Monitoring Windows Profiler Extensive Resource Monitoring Table 1. Tools fostering Performance Testing phases 2. Performance Testing Tools Evaluated The tools summarized in this section constitute a concise list of tools available over internet for automated performing testing (over a hundred tools were looked into). Each tool was evaluated based on the criteria defined in Section 3. 2.1 HP LoadRunner and Performance Center HP LoadRunner is a cost-effective and comprehensive tool used for automated performance testing. This is our first choice because: • LR is industry leader with over 90% market share • ChromeIS has most experience in HP LR/PC • Fully trained resources available • Uses standard programming techniques and common languages such as C, VB and Java; thus eliminating need to learn new language. • Easy to script, parameterize, correlate and close to real test execution • Extensive reporting, graphs, statistics collection, helping in root cause identification HP LoadRunner brought support for .net protocol in version 12 which was released last month (15 th April 2014); however, the support (as in readme file) does not describe component level details which can be recognized by LoadRunner. Through experiments, we can conclude that LR does not recognize OUR PRODUCT CMS client. It fails with the most basic level of scripting (login only – without selecting a branch) and the LoadRunner component i.e. VUGen stops responding. The script generated fails to compile and introduces number of errors. We can keep an eye on new patches of LR and keep experiments alive. However, this must not stop overall performance test progress. As of now, the tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. Copyrights © NetSol Technologies Ltd. All rights reserved 2 Unauthorized copy or use of this document is strictly prohibited
  9. 9. Software Quality Assurance Plan Performance Testing – Tool Selection 2.2 IBM Rational Performance Tester IBM RPT was 2nd choice since they hold nearly 6% out of 10% market share (after 90% being with LR). IBM RPT is slow in scripting, intelligence, execution and other aspects. However, nowhere on their website IBM claims that RPT supports .net or WCF or WPF. A formal conversation was opened with IBM but there had been no response. We can keep an eye on IBM newsletter to see if they provide any sophisticated mechanism for performance testing of WCF or SOA based architecture. As of now, the tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.3 Borland Silk Performer Silk Performer 15 now enables to capture all the raw measure data of a load test. It now automatically detects asynchronous communication patterns such as polling and long-polling. It more effectively supports applications with periodic, event-based, or server-triggered content updates. Tool is not as rich and introduced support for Ajax just recently with no support for modern needs of RIA based clients and smart clients. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.4 Telerik Test Studio The tool fails to recognize Telerik “ribbon” components. This was discussed with Telerik support and they confirmed the problem exists and listed as known issues. Since the tool fails to recognize the first component which loads after login, the tool cannot be used. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.5 Apache JMeter Through experiments, we can conclude the JMeter is able to script the calls, however, since OUR PRODUCT uses encryption, JMeter fails to decrypt the client/server communication. This is necessary since the JMeter cannot process an encrypted call. Ideally, we require JMeter to decrypt the server response, process it as required, re-crypt the newly formulated request and then send it to the server. JMeter needs to do it for each response from server to simulate load. Another workaround could be to remove encryption from OUR PRODUCT as long as application is undergoing performance testing. Although this imposes possible concern like: • The encryption and decryption is costly activity and requires additional layer of effort • The performance measures may not be realistic (slightly improved than real) since client and server don’t have to encrypt or decrypt Copyrights © NetSol Technologies Ltd. All rights reserved 3 Unauthorized copy or use of this document is strictly prohibited
  10. 10. Software Quality Assurance Plan Performance Testing – Tool Selection If the above constraint can be lived with, we can consider JMeter further. 2.6 j-Chav JChav is a way to see the change in performance of a web application over time, by running a benchmark test for each build. JChav reads all the JMeter logs from each run (one per build), and produces a set of charts for each test in each run. The tool compliments JMeter and hence cannot be right fit for OUR PRODUCT. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.7 Microsoft WCAT – Web Capacity Analysis Tool (formerly WAS – Web Application Stress Tool) WCAT (Web Capacity Analysis Tool) is an excellent tiny tool from Microsoft to perform load test web application on IIS. This tool enables users to do performance analysis on various scenarios of web application. All “perfmon” performance counters (like processor time, private bytes usage, disk queue length, total bytes sent or received in network) can be specified for the performance testing. Once scenario in application (i.e. appropriate URLs) is specified, a number of virtual clients from various client machines will visit the URL with appropriate request data. Like other performance tools, the following players are involved in a performance testing: • Server – machines on which your web application components are running • Client – A virtual client on a machine which acts as end-user to visit the URL • Controller – This coordinates a test among the virtual clients on various machines. It also captures and collates the performance counters from appropriate servers. • Works in Visual FoxPro. However, with all the excellence this tool may have, it is unable to recognize OUR PRODUCT application components. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.8 Sun VisualVM VisualVM is a free visual tool, originally from Sun, to monitor and troubleshoot Java applications. Runs on Sun JDK 6, but is able to monitor applications running on JDK 1.4 and higher. Utilizes various available technologies like jvmstat, JMX, the Serviceability Agent (SA), and the Attach API to get data and uses minimal overhead on monitored applications. Capabilities include: automatically detects and lists locally and remotely running Java applications; monitor application performance and memory consumption; profile application performance or analyze memory allocation; is able to save application configuration and runtime environment together with all taken thread dumps, heap dumps and profiler snaphots into a single application snapshot which can be later processed offline. Tool is more focused on java based applications. The tool does not address modern needs of RIA based clients and smart clients. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. Copyrights © NetSol Technologies Ltd. All rights reserved 4 Unauthorized copy or use of this document is strictly prohibited
  11. 11. Software Quality Assurance Plan Performance Testing – Tool Selection 2.9 WAPT WAPT can test and analyze the performance characteristics and bottlenecks of web under various load conditions with dynamic content and secure HTTPS pages. Real User Simulation with Dynamic Test Data, Recording and Playback of HTTPS/SSL Pages, Flexible Test Volume & Load Definition, Clear Reports and Graphs, Full Log of Virtual Users Activities Adjustable delays ("think times") between requests to simulate real user behavior Support of redirects, including redirects to another server Support of all types of proxies. The tool is very mature but it lacks support of WPF, WCF and smart clients. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.10 Web Application Load Simulator (LoadSim) LoadSim is a web application load simulator. It allows you to create simulations and have those simulations run against your webserver. It’s a web application load simulator. It allows you to create simulations and have those simulations run against your webserver. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.11 Neo Load NeoLoad is a professional load testing software for web applications (ASP.Net, J2EE, PHP, etc.) Works almost without scripting, automatic handling of ASP.Net VIEWSTATE, Automatic support of user sessions based on cookies or URL rewriting, Realistic VU simulation, cookies handling, plays simultaneously Virtual Users with different behaviors, modem emulation, IP spoofing, SSL support (record and playback), HTML/PDF/XML reporting, Compare results of several tests, Controller and Load Injectors are multi-platform: Windows, Unix. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. However, it successfully transcribes the sequence and composition of calls made to web services. This can be used in case the composition is not known to performance tester. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.12 EggPlant Load Tester eggPlant performance enabled HTTP-oriented Java and .NET Web Virtual Users (VUs) to be easily generated by recording use-cases. eggPlant Performance is designed for asynchronous requests, push technology, HTTP streaming, and dynamic data originated by the client by supporting multiple threads of execution and intelligent script generation. Some applications, however, simply cannot be automated at the HTTP level and are best automated at the GUI application level. In these situations eggPlant Performance can be used with eggPlant Functional or Selenium VUs for application level testing. eggPlant has in-built support for driving common non-web client-side APIs such as .NET and Java. This leverages the native support for security, session management, and data serialization making test creation faster, more effective, and more robust. The tool supports mobile clients, complex thick clients and propriety devices. More information is available. Copyrights © NetSol Technologies Ltd. All rights reserved 5 Unauthorized copy or use of this document is strictly prohibited
  12. 12. Software Quality Assurance Plan Performance Testing – Tool Selection The tool needs further RND to reach a conclusion. 2.13 OpenSTA – Open System Testing Architecture OpenSTA is a distributed software testing architecture based on CORBA. Using OpenSTA (Open System Testing Architecture) a user can generate realistic heavy loads simulating the activity of multiple concurrent virtual users. OpenSTA graphs both virtual user response times and resource utilization information from all Web Servers, Application Servers, Database Servers and Operating Platforms within environment, so that precise performance measurements can be gathered during load tests and analysis on these measurements can be performed. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.14 OpenWebLoad OpenWebLoad is a tool for load testing web applications. It aims to be easy to use and providing near real-time performance measurements of the application under test. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.15 OpenLoad OpenLoad provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.16 loadUI loadUI tool focused mainly on Light Directory Access Protocol (LDAP) and web applications working on HTTP, Web Services, REST, AMF, JMS, JDBC as well. The tool appears to be limited in functionalities it offers. The tool is incapable of handling Rich Internet Application (RIA) such as flex or Silverlight. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. 2.17 LoadeaTest LoadeaTest works in C# and .net and has various components. Capture module creates new test, modelize it as a script, enhances, replays and validates it. Control module defines test properties to launch test on several machines, monitors test in real time and gives a results view. An injection module executes script and collect data on different machines. Collect module is an http interface between all modules and database for storage dump. Last, analysis module is to view test results as tables and graphs, to generate reports, and to activate statistic algorithm on data (Principle Component Analysis). It works with HTTP, TCP, SOAP, RSS System resources. Scripts in high level language (C#, VB.NET ) and data in XML format. Copyrights © NetSol Technologies Ltd. All rights reserved 6 Unauthorized copy or use of this document is strictly prohibited
  13. 13. Software Quality Assurance Plan Performance Testing – Tool Selection The tool provides no mechanism for WCF specific applications in .net. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.18 LoadManager LoadManager was developed in Java (Eclipse platform). The result is a testing tool which runs on all platforms supported by Eclipse and Java such as Linux, Windows, HP Unix, etc. Supported protocols are: HTTP, HTTPS, Socket (TCP/IP), JMS, RMI, scripts in high level language (C#, VB.NET ) and data in XML format. Color-coding shows which users pass/fail and a script-link is provided* Testing of individual or groups of users can be paused/restarted at will. The tool provides no mechanism for WCF specific applications in .net. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.19 Test Automation FX TBC The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.20 CodedUI TBC The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.21 Soap UI TBC The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.22 Soap-Stone Soap Stone is a Network benchmark application which can put your network under load and conduct automatic benchmark and recording activities The tool is limited to network load. This is very basic to our requirement. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. Copyrights © NetSol Technologies Ltd. All rights reserved 7 Unauthorized copy or use of this document is strictly prohibited
  14. 14. Software Quality Assurance Plan Performance Testing – Tool Selection 2.23 Web Performance Suite Web Performance Suite includes a client-side analysis module (Analyzer) a load testing module (Load Tester) and server analysis modules. It is capable of handling most popular web-design technologies such as AJAX, Web forms, J2EE, ASP, .NET/C#, PHP, ColdFusion, Java, etc. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.24 Webserver Stress Tool Webserver Stress Tool is a HTTP-client/server test application The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.25 WebLOAD WebLOAD is an Open Source, available for free download and use, WebLOAD is a commercial- grade open source project with more than 250 engineering years of product development. Companies that require commercial support, additional productivity features and compatibility with third-party protocols have the option of purchasing WebLOAD Professional. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.26 QALoad QALoad is a performance testing tool from Micro Focus used to perform repeatable load testing and determine the ultimate performance and potential limits of a system QALoad has its own language, EasyScript. It has no support for Silverlight or Smart Client. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.27 Lisa Load LISA offers direct functional and performance validation of every component of distributed applications, with or without a UI present. LISA load tests also leverage the end-to-end functional tests created in LISA, invoking and verifying at every tier of the architecture, with a very low system footprint required, so capacity can be simulated at the lowest possible effort and cost. Lisa Load supports only Java/Swing; hence making it as not a choice for OUR PRODUCT. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. Copyrights © NetSol Technologies Ltd. All rights reserved 8 Unauthorized copy or use of this document is strictly prohibited
  15. 15. Software Quality Assurance Plan Performance Testing – Tool Selection 2.28 Grinder Grinder tool focused mainly on Light Directory Access Protocol (LDAP) and web applications working on HTTP protocol. The tool appears to be limited in functionalities it offers. The tool is incapable of handling Rich Internet Application (RIA) such as flex or Silverlight. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.29 nGrinder nGrinder is a stress testing tool, executing script creation, test execution, monitoring, and result report generator simultaneously. Since Grinder itself is not applicable to OUR PRODUCT, the GrinderStone is not applicable either. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.30 GrinderStone GrinderStone is an Eclipse plug-in for Grinder load testing scripts development including debugging, modularity and pretty logging. Since Grinder itself is not applicable to OUR PRODUCT, the GrinderStone is not applicable either. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.31 MStone Mstone started as a mail performance measurement system but now can simultaneously test SMTP, POP, IMAP, and some HTML based systems. It measures transaction latency in multiple stages, and graphs the combined results from multiple clients. MStone provider no mechanism for RIA or Smart clients. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.32 ValGrind Valgrind is focused on Linux programs. Built in tools automatically detect many memory management and threading bugs. ValGrind is not robust enough for RIA and smart clients. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.33 QEngine AdventNet QEngine WebTest provides accurate way to test the functionality of Web applications in both Windows and Linux machines. It provides an easy-to-learn user interface to completely automate the creation of sophisticated test scripts, allows record and playback Web browser events in IE and Mozilla browsers. The tool is immature and focused more on Regression automation with no support for Silverlight. Copyrights © NetSol Technologies Ltd. All rights reserved 9 Unauthorized copy or use of this document is strictly prohibited
  16. 16. Software Quality Assurance Plan Performance Testing – Tool Selection It is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.34 Qtest Qtest is automated load testing and monitoring solution, offering high capacity load simulation, extensive monitoring capability. Qtest supports all Web, Web Service, J2EE, .Net, ASP, AJAX, CGI, and Mainframe Portal environments and, in particular, complex Web applications like Siebel, SAP, and Epiphany. With its non-protocol based Winload module, Qtest also supports any non-web application. The tool offers sophisticated and complex frameworks, but nowhere does it provide Silverlight or Smart Client based on WCF. It is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.35 HammerHead 2 Hammerhead 2 is a stress testing tool designed to test out your web server by initiating multiple connections from IP aliases and simulated numerous (256+) users at any given time. The rate at which Hammerhead 2 attempts to pound web server site is fully configurable. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.36 Hammerora Hammerora is a load generation tool for the Oracle Database and Web Applications. Hammerora includes pre-built schema creation and load tests based on the industry standard TPC-C and TPC-H benchmarks to deploy against the Oracle database with multiple users. Hammerora also converts and replays Oracle trace files and enables Web-tier testing to build bespoke load tests for your entire Oracle application environment. The tool provides no mechanism to test web services or RIA based or desktop based smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.37 Pylot Pylot is open performance and scalability testing tool, capable of running HTTP load tests to compliment capacity planning, benchmarking, analysis, and system tuning. Pylot generates concurrent load (HTTP Requests), verifies server responses, and produces reports with metrics. Tests suites are executed and monitored from a GUI. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.38 OpenWebLoad preVue-X provides non-intrusive regression and performance testing of X-Window applications. preVue-X accurately emulates multiple users while requiring no special hooks into your application Copyrights © NetSol Technologies Ltd. All rights reserved 10 Unauthorized copy or use of this document is strictly prohibited
  17. 17. Software Quality Assurance Plan Performance Testing – Tool Selection or X Libraries, thus guaranteeing that the application tested is the same application that will be deployed. preVue-X operates at the X protocol level and independently of the GUI. It automatically collects all user interactions with the application (e.g., mouse movements, button activity, keystrokes, think time delays) into an easily editable test script for playback. Features include automated synchronization and time-stamping; geometry independence and script viewer The tool does not speak about WCF in specific or smart clients. Need further RND to reach a conclusion. 2.39 SD’s Profiler SD's Profiler collects information about execution, frequency of all blocks of code in an application. A graphical display sorts this information by file or by frequency, and the relative execution costs are shown graphically superimposed over the source code. Available for C, COBOL, and Java. For C and Java, easily customized to work in embedded or real-time contexts. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.40 Benerator benerator is a framework for creating realistic and valid high-volume test data, used for (unit/integration/load) testing. Metadata constraints are imported from systems and/or configuration files. Data can be imported from and exported to files and systems, anonymized or generated from scratch. Domain packages provide reusable generators for creating domain- specific data as names and addresses internationalizable in language and region. It is customizable with plugins and configuration options. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.41 SOATest (by Parasoft) Parasoft SOA Quality Solution continuously measure how each service conforms to the often dynamic expectations defined by both your own organization and your partners, reduces the complexity of testing in today’s heterogeneous environment – with limited visibility/control of distributed components or vendor-specific technologies. It validates all critical aspects of complex transactions, which may extend beyond the message layer through a web interface, ESBs, etc. The tool provides no clarity in order to test WCF or smart client or web services. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.42 Scapa Test and Performance Platform Scapa Test and Performance Platform is testing, diagnosis and monitoring product. The Platform works by simulating multiple users of a computer system. By measuring the performance seen by virtual users, it is possible to understand how the system would behave under a corresponding level of real usage. Copyrights © NetSol Technologies Ltd. All rights reserved 11 Unauthorized copy or use of this document is strictly prohibited
  18. 18. Software Quality Assurance Plan Performance Testing – Tool Selection The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.43 CLIF Load Injection CLIF is a modular and flexible distributed load testing platform. It may address target systems reachable from a Java program (HTTP, DNS, TCP/IP...) CLIF provides 3 user interfaces (Swing or Eclipse GUI, command line) to deploy, control and monitor a set of distributed load injectors and resource consumption probes (CPU, memory...) An Eclipse wizard helps programming support for new protocols. Load scenarios are defined through XML-editing, using a GUI, or using a capture tool. The scenario execution engine allows the execution of up to millions of virtual users per load injector. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.44 ContiPerf ContiPerf is a small testing utility that enables the user to easily leverage JUnit 4 test cases as performance tests e.g. for continuous performance testing. It is inspired by JUnit 4's easy test configuration with annotations and by JUnitPerf's idea of wrapping Unit tests for performance testing, but more powerful and easier to use. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.45 Coveriety TBC The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.46 D-ITG (Distributed Internet Traffic Generator) D-ITG (Distributed Internet Traffic Generator) is a platform capable to produce traffic at packet level accurately replicating appropriate stochastic processes for both IDT (Inter Departure Time) and PS (Packet Size) random variables. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. Copyrights © NetSol Technologies Ltd. All rights reserved 12 Unauthorized copy or use of this document is strictly prohibited
  19. 19. Software Quality Assurance Plan Performance Testing – Tool Selection 2.47 Database Opensource Test Suit (DOTS) The Database Opensource Test Suite (DOTS) is a set of test cases designed for the purpose of stress- testing database server systems in order to measure database server performance and reliability. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.48 DB Monster DBMonster is an application to generate random data for testing SQL database driven applications under heavy load. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.49 Deluge Deluge is an open-source web site stress test tool. It simulates multiple user types and counts, includes proxy server for recording playback scripts, and log evaluator for generating result statistics. This tool has not been updated since 2002. This tool is no longer under active development although it is still available on Sourceforge. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.50 DieselTest Dieseltest is a Windows application that simulates hundreds or thousands of users hitting a website. BEWARE: This tool has not been updated since 2001. It remains listed here in case anybody wishes to take it over. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.51 Faban Faban is a facility for developing and running benchmarks, developed by Sun. It has two major components, the Faban harness and the Faban driver framework. The Faban harness is a harness to automate running of server benchmarks as well as a container to host benchmarks allowing new benchmarks to be deployed in a rapid manner. Faban provides a web interface to launch & queue runs, and extensive functionality to view, compare and extract graph of outputs. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. Copyrights © NetSol Technologies Ltd. All rights reserved 13 Unauthorized copy or use of this document is strictly prohibited
  20. 20. Software Quality Assurance Plan Performance Testing – Tool Selection The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.52 FunkLoad FunkLoad is a functional and web performance testing tool, written in Python. It can be used for functional and regression testing of web projects as well as performance testing and stress testing to overwhelm the web application resources and test the application recoverability. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.53 Curl-Loader Written in C, Curl-Loader is a web application testing and load generating tool. Provides an open- source alternative to Spirent Avalanche and IXIA IxLoad. The loader uses real HTTP, FTP and TLS/SSL protocol stacks, simulating tens of thousand and hundred users/clients each with own IP-address. The tool supports user authentication, login and a range of statistics. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.54 Spirant Avalanche TBC. 2.55 IXIA IxLoad TBC. 2.56 Paessler TBC The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.57 AQ TestComplete and AQ Time AQTime is a performance profiler and memory allocation debugger for 32-bit and 64-bit Windows and .NET-connected applications. AQtime can be integrated into Microsoft Visual Studio, Borland Developer Studio or CodeGear RAD Studio or CodeGear RAD Studio or it can run as a stand-alone application. AQtime can also determine what caused bottlenecks. The tool provides no mechanism to test RIA based applications or WCF calls. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. Copyrights © NetSol Technologies Ltd. All rights reserved 14 Unauthorized copy or use of this document is strictly prohibited
  21. 21. Software Quality Assurance Plan Performance Testing – Tool Selection 2.58 Test Maker TestMaker offers building and running intelligent test agents that test Web-enabled applications for scalability, functionality, and performance. It works with object-oriented scripting language (Jython) to build intelligent test agents, an extensible library of protocol handlers (HTTP, HTTPS, SOAP, XML-RPC, SMTP, POP3, IMAP), a new agent wizard featuring an Agent Recorder to write scripts, a library of fully-functional sample test agents, and shell scripts to run test agents from the command line and from unit test utilities. The tool provides no mechanism to test RIA and smart client. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.59 TP Test TPTEST measures the throughput speed to and from various reference servers on the Internet. The use of TPTEST may help increase the consumer/end user knowledge of how Internet services work. The tool is very basic and cannot be graded as performance testing tool for applications. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.60 Tsung Tsung is a distributed load testing tool. It is protocol-independent and can currently be used to stress HTTP, SOAP and Jabber servers (SSL is supported). It simulates complex user's behavior using an XML description file, reports many measurements in real time (including response times, CPU and memory usage from servers, customized transactions, etc.). HTML reports (with graphics) can be generated during the load. For HTTP, it supports 1.0 and 1.1, has a proxy mode to record sessions, supports GET and POST methods, Cookies, and Basic WWW-authentication. It has already been used to simulate thousands of virtual users. The tool requires further RND to experiment with OUR PRODUCT POS. 2.61 Tnsgen Tnsgen is test data generation tool which operates under Solaris, Linux and Windows, capable of generating thousands of records per second in XML, SQL or user-defined formats. Easily extensible to include project-specific data sets. The tool is a data generating tool, it cannot be graded as performance testing tool. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.62 N-Time N-Time tool is very similar to NUnit tool to perform repeatable tasks that help managers, architects, developers and testers to test an application against its performance. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. Copyrights © NetSol Technologies Ltd. All rights reserved 15 Unauthorized copy or use of this document is strictly prohibited
  22. 22. Software Quality Assurance Plan Performance Testing – Tool Selection 2.63 p-unit p-unit is an open source framework for unit test and performance benchmark, under GPL license. p-unit supports to run the same tests with single thread or multi-threads, tracks memory and time consumption, and generates the result in the form of plain text, image or pdf file. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.64 Lobo Lobo tool supports performance testing of Java based applications only. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.65 MessAdmin MessAdmin is a light-weight and non-intrusive notification system and HttpSession administration for J2EE Web Applications, giving detailed statistics and informations on the application. It installs as a plug-in to any Java EE WebApp, and requires zero-code modification. The tool is specific for java based application. It provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.66 Multi Router Traffic Grapher (MRTR) Multi Router Traffic Grapher is written in perl and works on Unix/Linux as well as Windows and even Netware systems. MRTG is free software licensed under the Gnu GPL. The tool is specific for java based application. It provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.67 RRRD Tool RRRD Tool is data logging and graphing system for time series data. It can be used to write custom monitoring shell scripts or create whole applications using its Perl, Python, Ruby, TCL or PHP bindings. The tool is barely into application performance testing. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.68 Multi-Mechanize Multi-Mechanize is an open source framework for performance testing. It allows running simultaneous python scripts to generate load against a web site or web service. Copyrights © NetSol Technologies Ltd. All rights reserved 16 Unauthorized copy or use of this document is strictly prohibited
  23. 23. Software Quality Assurance Plan Performance Testing – Tool Selection The tool does not address modern needs of RIA based clients and smart clients. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.69 Stress Driver Stress Driver is a general-purpose stress test tool. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.70 Load Impact LoadImpact is online load testing service from loadimpact.com for load- and stress- testing; with access to distributed network of load generator nodes – server clusters with very fast connections to enable simulation of simultaneous users accessing your website concurrently. Free low level load tests for 1-50 simulated users; higher levels have monthly fees. No claim to test RIA client on their site. No support for smart client. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.71 Pure Load PureLoad is a load testing tool that simulates concurrent users executing requests against server based applications. Use PureLoad to verify performance and stability of your server applications. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.72 Pure Agent PureAgent is a performance monitoring tool that replicates real users generating activities on server applications. PureAgent measures the total round trip time it takes to execute one or many user operations accessing either static or dynamic content. The results are presented as charts in a standard web browser, alarm levels can be defined which might trigger notifications to be sent by Email. The monitored scenario of user operations is created using the PureTest application. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.73 FWPTT fwptt is used for load testing of web applications only. It can record normal and ajax requests. It can understand HTML being returned from server, hence, it works with asp.net, jsp, php or other such applications. The tool recently supported AJAX as well. Since AJAX is the extent of the tool, the tool is too basic to meet the complex requirements of OUR PRODUCT where web is using silverlight and smart client. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. Copyrights © NetSol Technologies Ltd. All rights reserved 17 Unauthorized copy or use of this document is strictly prohibited
  24. 24. Software Quality Assurance Plan Performance Testing – Tool Selection 2.74 JBlits Pro JBlits gives continuous heavy loading and automated response validation. Easy to use, quick to setup. Fully featured with logging, graphs, stats, scheduling, HTTP debugger etc. Full Java API. Works in Java, works for Java. The tool is not meant for Silverlight and smart client. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.75 JCrawler JCrawler can start with a set of starting URLs and it will begin crawling from that point onwards, going through any URLs it can find on its way and generating load on the web application. Although, the load parameters (hits/sec) are configurable, JCrawler doesn’t look like a real performance testing tool. There are no built-in mechanism for correlation and parameterization. The tool, at best, can simulate load on static websites where no custom data is required. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.76 PandoraFMS Pandora FMS is a monitoring Open Source software and watches systems and applications. Pandora FMS could detect a network interface down, defacement in website, a memory leak in server application, or the movement of any value of the NASDAQ new technology market. Pandora FMS could send out SMS message when system fails... or when Google's value drop below US$ 500. Although PandoraFMS provides features unique to itself and it can be very useful in many ways, PandoraFMS fails to simulate load on RIA and Smart Client. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.77 Siege Siege can be used to test over HTTP and HTTPS. SIEGE is an http regression testing and benchmarking utility, designed to measure the performance of code. It lets the user hit a webserver with a configurable number of concurrent simulated users. SCOUT surveys a webserver and prepares the urls.txt file for a siege. In order to perform regression testing, siege loads URLs from a file and runs through them sequentially or randomly. Scout makes the process of populating that file easier. You should send out the scout, before you lay siege. With the above limitation of the tool, Siege is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.78 HTTP Load http_load is a multiprocessing http test client. As the name suggests, the tool works specifically for HTTP protocol. This helps cater plain HTML applications only. http_load runs multiple http fetches in parallel, to test the throughput of a web server in a single process. It can be configured to do https fetches as well. It processes a file containing a list of URLs that may be fetched, a flag specifying how to start connections (either by rate or by number of simulated users), and a flag specifying when to quit (either after a given number of fetches or a Copyrights © NetSol Technologies Ltd. All rights reserved 18 Unauthorized copy or use of this document is strictly prohibited
  25. 25. Software Quality Assurance Plan Performance Testing – Tool Selection given elapsed time). There are also optional flags for checksums, throttling, random jitter, and progress reports. With all above, the tool serves very basic level requirement for OUR PRODUCT. Provides no mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.79 httPerf httperf provides facility for generating HTTP workloads. The three distinguishing characteristics of httperf are: • its robustness, which includes the ability to generate and sustain server overload, • support for the HTTP/1.1 and SSL protocols, and • its extensibility. With all above, the tool serves very basic level requirement for OUR PRODUCT. Provides no mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.80 iPerf iperf was developed as a modern alternative for measuring maximum TCP and UDP bandwidth performance. iperf allows the tuning of various parameters and UDP characteristics. iperf reports bandwidth, delay jitter, datagram loss. With all above, the tool serves very basic level requirement for OUR PRODUCT. Provides no mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.81 Site Blaster Site Blaster is a web site load and stress testing tool. It can be used to rapidly submit requests to a site. Or, it can pause a random amount of time between submissions; approximating the behavior of a user. While the testing is being performed, the pages being tested will be displayed. When testing is complete, a report is available that can be viewed or printed. It provides no mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.82 RawLoadTester RLT calls the selected URL multiple times and tells server response time. It writes some additional runtime details to the PHP log file to optionally do more granular analysis afterwards. Although the server processes most of the statistics, all URL requests come from the browser. You can run as many browsers and workstations simultaneously as you want. Copyrights © NetSol Technologies Ltd. All rights reserved 19 Unauthorized copy or use of this document is strictly prohibited
  26. 26. Software Quality Assurance Plan Performance Testing – Tool Selection RLT provides no mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.83 Allmon Allmon is an open source, generic system collecting and storing metrics used for performance and availability monitoring. It provides no mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.84 AppsWatch Appswatch is an automated testing tool for performance, and GUI testing and SLA monitoring and alerting. It is protocol independent and works with any application accessible from windows desktop. It works for Citrix, Java, .NET, Oracle, SAP, Eclipsys, Mckessen, Meditech, GE Healthcare, Epic, Cerner, Client-Server, Flex, Flash and even custom applications It provides no mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.85 BurnIn Test BurnInTest allows sub-systems of a computer to be simultaneously stress tested for endurance, reliability and stability. BurnInTest tests the CPU, hard drives, CD ROMs, DVDs, sound cards, 2D graphics, 3D graphics, RAM, network connections & printers. The Pro version can also test tape drives, USB, serial and parallel ports. This can all be done simultaneously as the application is multithreaded. It provides no mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.86 Capacity Calibration (CapCal) CapCal writes code in Java for measuring web site performance under various loads--by actually generating those loads, in real time, from computers scattered all across the Internet. CapCal can also be distributed across your LAN and WAN, giving realistic performance data that matches end user's experience. It provides no mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. Copyrights © NetSol Technologies Ltd. All rights reserved 20 Unauthorized copy or use of this document is strictly prohibited
  27. 27. Software Quality Assurance Plan Performance Testing – Tool Selection 2.87 Chariot Chariot tests and troubleshoots any segment of network and provides comprehensive reports of the results by emulating transaction traffic from real applications. It can be used to predict how well network infrastructure will handle popular applications prior to deployment, and to isolate and evaluate the performance of new network devices. Chariot is highly scalable and can be used to set up and control tests—emulating up to 10,000 connections and hundreds of thousands of end- users—all from a central console. It cannot be graded as real performance testing tool since it aims for network and infrastructure against internet applications. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.88 CitraTestVU CitraTestVU is a load testing tools, works in Visual Basic scripting to simulate Load on Citrix & Microsoft Terminal Services Hosted Environments. The tool is focused on Java only. It provides no mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.89 SSW Performance Pro SSW Performance PRO! isolates performance bottle-necks in your Access 97, 2000, 2002 or 2003 app. It checks every form, combo and list box, and reviews SQL statements. It reports exactly what to fix to improve the speed of your application. It’s especially useful after you've upsized an application to SQL Server to pinpoint black spots in your app. The tool is very basic and mainly targets MS Access. It cannot be graded a commercial level tool. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.90 e-Load e-Load is enables accurate testing of the response times and scalability of web applications and web services. e-Load can concurrent users, executing real business transactions to analyze how well web software applications will perform under load. The tool provides no mechanism to test RIA based and desktop based applications. It supports Web Services but not WCF. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.91 MLoad MLoad generates high levels of realistic network and user traffic ensuring that WAP Gateways, MMSCs and web based mobile infrastructure elements meet the high-bandwidth, high- performance demands of a mobile internet service. The toolset includes a test harness to allow acceptance testing of mobile elements against performance requirements in the deployed environment. The tool focused on data flux and network bandwidth only. It provides no mechanism to test RIA based and desktop based applications. It supports Web Services but not WCF. Copyrights © NetSol Technologies Ltd. All rights reserved 21 Unauthorized copy or use of this document is strictly prohibited
  28. 28. Software Quality Assurance Plan Performance Testing – Tool Selection The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.92 Monitor Master Monitor Master is a quality management solution for wireless data services - WML/XHTML, SMS and MMS. Monitor Master helps Operators and Content Developers ensure that end-users of wireless applications will have the best quality of experience possible - regardless of the handset being used. The tool focuses wireless protocols and not RIA based and desktop based applications. It supports Web Services but not WCF. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.93 Forecast Forecast is a easily and accurately tests the performance and scalability of IT applications and systems. No support for WPF, WCF or Smart Client. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.94 GS DataGenerator GS DataGenerator generates high volumes of 'intelligent' test data on virtually any relational database, creates business trends and patterns, simulates business workflow, mimics potential data errors Employ number of predefined functions. This broad range of functionality allows testing database performance and functionality during development. It’s very basic tool, works in DOS and provides no support for WPF, WCF or Smart Client. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.95 vPerformer vPerformer is used to assess the performance and scalability of web applications. vPerformer allows assesses how web application responds when it is concurrently accessed by a large number of users. It can measure the performance characteristics of application by recording and replaying automated scripts that simulate a large number of concurrent virtual users. The tool is very basic and doesn’t address modern needs like Silverlight etc. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.96 JUnit JUnitPerf is a collection of JUnit test decorators used to measure the performance and scalability of functionality contained within existing JUnit tests The tool is specific for Java related apps. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. Copyrights © NetSol Technologies Ltd. All rights reserved 22 Unauthorized copy or use of this document is strictly prohibited
  29. 29. Software Quality Assurance Plan Performance Testing – Tool Selection 2.97 Xceptance LoadTest Xceptance LoadTest for web and Java and other apps includes recording capabilities. XLT Cloud Service available. Tests implemented as JUnit 4 test cases. For web-based tests, the framework provides a (headless) browser that can emulate Internet Explorer or Firefox behavior. Can execute client-side JavaScript in the emulated web browsers and that way it simplifies the creation of test cases for Web 2.0 applications. It is platform independent since the tool is implemented in Java; test scripting in Java or Ruby. The tool is free for up to five virtual users. The tool is focused on Java only. It provides no mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.98 zapwireless zapwireless is a wireless throughput test tool which can be used to fully characterize the statistical performance of a wireless link. It provides no mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.99 Seagull Seagull was primary aimed at IMS protocols, now it is a traffic generator for functional, load, endurance, stress and performance tests for almost any kind of protocol. Currently supports Diameter, XCAP over HTTP, TCAP (GSM Camel, MAP, Win) protocols. It provides no mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.100 SIPp SIPp is a performance testing tool for the SIP protocol. Its main features are basic SIPStone scenarios, TCP/UDP transport, customizable (xml based) scenarios, dynamic adjustement of call- rate and a comprehensive set of real-time statistics. It can also generate media (RTP) traffic for audio and video calls. Although Sipp is a rich tool and may be useful for specific purposes, it provides no mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.101 SLAMD SLAMD Distributed Load Generation Engine is a Java-based application designed for stress testing and performance analysis of network-based applications. SLAMD does not provide any mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. Copyrights © NetSol Technologies Ltd. All rights reserved 23 Unauthorized copy or use of this document is strictly prohibited
  30. 30. Software Quality Assurance Plan Performance Testing – Tool Selection The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.102 IxoraRMS IxoraRMS is a monitoring tool with great visualization and customization capabilities. But it severely lacks capabilities to support RIA and modern complex architecture. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.103 j-hawk j-hawk is a Java based open source framework which can be incorporated within an application for performance testing. The idea requires to define *module and its method inside the application and register the same with j-Hawk. j-Hawk executes the modules and generates a graphical performance report which can be analyzed to find performance bottleneck of your application. The tool is specific for Java based application which is not the case with OUR PRODUCT. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.104 Wireshark TBC The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.105 Ostinato Ostinato is an open-source, cross-platform packet/traffic generator and analyzer with a friendly GUI. It aims to be "Wireshark in Reverse" and thus become complementary to Wireshark.The tool is specific for Java based application which is not the case with OUR PRODUCT. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.106 Eclipse Test & Performance Tools Platform Project (EPTP) EPTP supplies frameworks and services that allow software developers to build unique test and performance tools, both open source and commercial, that can be easily integrated with the platform and with other tools. Eclipse Test & Performance Tools Platform Project (TPTP) evidently provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.107 Web Polygraph Web Polygraph is a benchmarking tool for caching proxies, origin server accelerators, L4/7 switches, content filters, and other Web intermediaries. This tool focused mainly on Light Directory Access Protocol (LDAP) and web applications working on HTTP protocol. The tool appears to be limited in functionalities it offers. The tool is incapable of handling Rich Internet Application (RIA) such as flex or Silverlight. Copyrights © NetSol Technologies Ltd. All rights reserved 24 Unauthorized copy or use of this document is strictly prohibited
  31. 31. Software Quality Assurance Plan Performance Testing – Tool Selection The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 3. Tools Selection Criteria Criteria for selection of Automated Performance Testing tool was defined before proceeding with research activities. The criteria broadly address Business and Technical Requirements and Objectives sought from Performance Testing. Below is a highlight of foundation of criteria: 3.1 Business and Technical Requirements 1. It is essential for the application is able to pass at least 1 test case to perform certain business process. Although, it is best to have reasonably stable application before performance testing starts. If there is urgency, the application should be able to work under at least 1 situation/record-set. 2. It is pivotal to finalize performance test objectives before test scripting is begun. Number of users, data volume and other variables must be known beforehand otherwise unnecessary delays may occur. For example, if the intention is to run endurance or longevity test with 1000 users over 4.5M record base, for 24 hours; then test data requirement will be massive as compared to running 100 users for 1 hour. 3. Dedicated test environment and interface systems are identified and requisition (booking) made in advance and certified to be used for testing. The hardware specifications should be documented for web server, application server, db server and any other layer involved. This should be documented for Load Injectors/Generators as well. 4. System components have been identified and deployed over test environment mimicking (or a close match to) production server. 5. Test IDs/passwords and access rights have been identified, requisition made and approval granted for all involved applications. Test IDs must have the correct access rights for testing for application as required by user’s role. 6. Criteria for test data to be used for testing has been drawn, and reviewed by respective parties involved. 7. Test Data has been created in testing database. 8. System infrastructure and connectivity test are performed prior to each test phase to ensure connectivity is enabled. 3.2 Performance Test Objectives In general, the test objective is to ensure the respective applications meet the stated and implied performance requirements of business; as well as, the statistics remain within the SLA agreed with end customer. Copyrights © NetSol Technologies Ltd. All rights reserved 25 Unauthorized copy or use of this document is strictly prohibited
  32. 32. Software Quality Assurance Plan Performance Testing – Tool Selection 1. Identify any bottlenecks and slow system performance 2. Ensure application response time is acceptable with data volume of 4.5 Million records 3. Gauge application responsiveness under Stress 4. Validate application behavior under endurance and longevity (24+ hours) 5. Validate the behavior of Load balancing over cluster nodes equally 6. To ensure that application is functioning properly under load 7. Establish performance benchmarks 8. Validate transactions response time against customers’ SLA 9. Perform root cause analysis for any problems and errors appearing under load – and suggest possible optimization. 10. Draw projection as measures for future needs 11. Ensure smooth implementations with no/minimum disruption to business going live, and applications rollout with minimum risk. 12. Project implementation compliance to ChromeIS processes. 13. Project quality that meets entry and exit criteria for each performance testing phase. There are specific goals of performance testing, like: 1. Measure and benchmark response time for each page/screen to open 2. Measure and benchmark response time against each transaction, like, how long it takes when, saving a proposal, saving a contract, converting a proposal into contract, calculating exposure of a customer, asset is attached to a proposal, rentals are generated, financial analysis is done, early termination is performed etc. 3. Measure and benchmark response time for each service as an individual. 4. This section specifies the test types (with definition for each) to run. This also explains when application will be considered fit to undergo each type of testing and when a particular 5. Measure and benchmark maximum number of users a transaction/service can entertain disregarding response time (peak point identification) 6. Benchmark maximum number of users achieved against test environment (hardware/software specifications) 7. Benchmark maximum hardware utilization against desired number of users (against test environment specifications) 8. Benchmark system behavior against cold startup and warm startup 9. The spike ratio that the system must be capable of handling 10. The peak ratio that the system must be capable of handling? 11. Memory leakage against each business process or individual service Copyrights © NetSol Technologies Ltd. All rights reserved 26 Unauthorized copy or use of this document is strictly prohibited
  33. 33. Software Quality Assurance Plan Performance Testing – Tool Selection 12. Identify which SPs fail and at what circumstances, at what line, under what conditions. 13. Facilitate in ensuring load is distributed evenly over all nodes of clusters (as configured) 14. Facilitate ensure seamless mitigation of failover under high load 15. Tolerance ratio: Imposed load ? 25 %? 16. Spike ratio: Imposed load x 3? 17. Burstiness ratio: Imposed load x 5? 18. Increase the load by multiplying the load baseline by 1x, 2x, 3x, 4x, Nx gradually until unacceptable response time is reached through optimization. 19. Pass recommendation in achieving optimal configuration, or identifying target component where optimization could be required. 0.1 Second 10 Second 20 Second 30 Second Acceptable User Experience Unacceptable User Experience Questionable User Experience Business is Closed Response Time Load 60 Second Figure 1, Performance Acceptance at a glance The above assumes, all stakeholders are on same page regarding: • What is transaction response time and what is acceptable response time? • The difference between simultaneous and concurrent users • Which metrics should be collected during all types of performance testing? • What is the correlation between demand and increased load? Copyrights © NetSol Technologies Ltd. All rights reserved 27 Unauthorized copy or use of this document is strictly prohibited
  34. 34. Software Quality Assurance Plan Performance Testing – Tool Selection • How do we determine which components are problematic? 3.3 Performance Test Execution Performance testing usually follows a linear path of events: • Create performance-testing scripts and scenarios. • Create Task Distribution Diagram to elaborate maximum number of users over a component as well as maximum number of users accessing system at any point of time. • Execute performance tests as standalone tests to detect issues within a particular business workflow. Repeat execution multiple times to surface most errors, identify root cause of failure. • Execute performance scenarios as a package (i.e. with all BPs together) to simulate real life activities that are measured against performance success criteria. • Monitor all variables involved in Performance Testing. This includes all servers and their components such as processor, memory, NIC, clusters, caching, heap, top SQL queries, SPs, ISS caching, IIS sessions, Kernel, Number of processes, Number of threads, Throughput, DB Sessions, Parameters, Data Buffer Cache Hit Ratio, Shared Pool Hit Ratio, Query to Parse ratio, PGA Usage, I/O read wait time, I/O write wait time, CPU wait time, configurations, hits per second, timeouts etc. These statistics will be collected for Web Server, App Server, DB Server and any other server involved, as well as load injectors. • Collate all statistics and observations. Perform analysis and document recommendations/suggestions. Compile a concise performance testing results report. • Tune the system. • Repeat testing as required. 3.4 Load Description In the context of Performance Testing, “Load” can be broadly classified into 2 categories, namely, Server Based Load and User Based Load. Below is the elicitation of both terminologies. Server-based Load • Number of users and/or sessions • Average session time • Number of page views • Average page views per session • Peak period (e.g., 75% of traffic is from 11:00 AM-4:00 PM) • Number of hits • Average page size • Most requested pages • Average time spend on page Copyrights © NetSol Technologies Ltd. All rights reserved 28 Unauthorized copy or use of this document is strictly prohibited
  35. 35. Software Quality Assurance Plan Performance Testing – Tool Selection • Frequency of visits (e.g., 75% of users made one visit) • Demographics • Client information such as browser, browser version, Java script support, Java script enable/disable, and so on. User Based Load: • Number of users • Session length • User activities and frequency of activities per session • Think/Read/Data-input time • Percentage by functional group • Percentage by human speed • Percentage by domain expertise/ familiarity (speed) • Percentage by demographics (arrival rates) Other questions to consider when describing system Load: • What is the definition of “workload”? • How do we size the workload? • What is the expected workload? • What’s the mix ratio of static pages vs. code? • What is the definition of “increased load”? • What is future growth? Can it be quantified? • What is the definition of scalability? A requirements document is filled up for each testing cycles to further clarify performance test requirements. Refer to below example file. 4. Conclusion After assessing large number of tools available in market, we can conclude that HP LoadRunner, IBM Rational Performance Center and JMeter (the top 3 market leaders) do not directly support testing the complex architecture we’re building; at least not as yet. A number of open source tools, freeware utilities, ASM based performance testing services and other freeware utilities do not address the performance testing challenges we‘re facing. JMeter could be a possible option if encryption can be removed from OUR PRODUCT for testing purposes. EggPlant Load Tester described in detail how it can test web services written in .net. This requires further RND and experimentation if time allows. Tsung is an open source framework which claims testing web services and other complex niche technology frameworks. This requires further RND and experimentation if time allows. We’ve had trial runs with Visual Studio and optimistically, we can hope to meet our objectives with MSVS. This is further described in Section 4.1 below. Copyrights © NetSol Technologies Ltd. All rights reserved 29 Unauthorized copy or use of this document is strictly prohibited
  36. 36. Software Quality Assurance Plan Performance Testing – Tool Selection 4.1 Selected Tools The testing tool chosen is Microsoft Visual Studio 2013. NeoLoad can be used to identify the sequence of calls made to web services. Although, MSVS this is not specifically designed performance testing in mind, however, this has been picked as a workaround for current situation. Visual Studio offers scripting capabilities, thus enabling to simulate asynchronous multithreaded simultaneous users to the application. This is the first stepping stone for us. Once the load is simulated closed to real behavior, various other tools will be used to identify application behavior and other environmental factors undergoing changes (as described in Section 3) Below table enlists tools which will be used for Performance Testing as well as other activities (e.g. resources monitoring) during performance testing phase: Tool Purpose Visual Studio 2013 For Performance Testing of areas which can’t be worked out with HP PC / LR. HP LoadRunner 12 Automated performance and stress testing NeoLoad Trace client/server communication SolarWind Defects logging and tracking Spotlight Enterprise Resource Monitoring Windows Profiler Extensive Resource Monitoring Table 2. Selected tools for Performance Testing phases 4.2 SWOT Analysis In concluding this research, we’ve learned about numerous testing tools, what those are capable of and their approach towards load simulation. HP LoadRunner is a market leader in performance testing tools, with over 90% market share globally. ChromeIS has mature strong expertise in HP LoadRunner and HP Performance Center and that had always been strength of ChromeIS to address any performance concerns in timely fashion. Unfortunately, HP LoadRunner has very limited support for .net protocol and it induces a lot of errors whilst preparing performance test scripts for OUR PRODUCT. Experiments show that the tool fails in deserialization, decoding and parsing already desterilized fragments of scripts. We can conclude that LoadRunner 12 does not offer what it takes to test a sophisticated .net based application with complex architecture and modern clients. Time has changed and ChromeIS has taken initiatives to re-invent the products using state of the art technologies making Rich Internet Applications and optimizing performance using Smart Client. While this sounds exciting, it brings unforeseen challenges in many ways. Performance and efficiency of software is undoubtedly among most critical aspects of software and this cannot be ascertained without undergoing aggressive performance test. Hence, a workaround has been suggested and Microsoft Visual Studio 2013 will be used for conducting performance testing. Copyrights © NetSol Technologies Ltd. All rights reserved 30 Unauthorized copy or use of this document is strictly prohibited
  37. 37. Software Quality Assurance Plan Performance Testing – Tool Selection 4.3 Training Needs ChromeIS has mature expertise in HP LoadRunner and HP Performance Center; however, these tools are out of option. Since ChromeIS will be using Visual Studio 2013 for development of Performance test agents, certain level of resource training is required. A team size of 4-6 dedicated resources is expected to be working for Performance Testing. This team will be briefly trained by QAM on performance testing goals, objectives and overall approach towards meeting quantitative goals. While the conceptual and quantitative goals remain the same, working with the tool itself will be crucial to achieve test objectives. Training programs to be provided to the team to enable successful planning and execution. A training plan can be published separately or as an addendum to this document. 4.4 Lessons Learnt Selection of tool and technology for carrying out performance testing is of utmost significance. The same applies to overall software development as well. We’ve seen in past as well that development technology was picked without consensus, or even consultation with Performance Testing team. Now we’ve understood that performance testing an application involves sophisticated activities, induces unforeseen technical challenges, and imposes technological constraints, thus essentially making it a challenge to workout performance testing. This cannot be simply compared with business testing where there is no or little involvement of technological constrains. It is suggested to keep performance testing team in due consultation when making technology selection or making technical changes to existing application (upgrade of Silverlight version, bringing parts of application logic to thick client). Although the final decision of picking the development technology or modifying underlying components will be under sole discretion of the PM, this discussion will ensure any possible constraints are communicated and any possible risks are addressed 5. Definition of Terms, Acronyms And Abbreviations Term Description DEV Development Team PT Performance Testing SQA Software Quality Assurance Copyrights © NetSol Technologies Ltd. All rights reserved 31 Unauthorized copy or use of this document is strictly prohibited

×