SlideShare a Scribd company logo
Software Quality Assurance Plan
For WCF/WPF based Application
Version 0.1
May 15, 2014
Compiled by:
Kamran Shaukat Ali Khan
Chrome Technologies
ChromeIS.com
Software Quality Assurance Plan
Performance Test Plan – Ascent –Version 0.1
Table of Contents
Document Information................................................................................................vi
Document Information.................................................................................................vi
Document Review Information.....................................................................................vi
Document Review Information.....................................................................................vi
Document Revision History..........................................................................................vi
Document Revision History...........................................................................................vi
List of Figures..........................................................................................................vii
List of Figures.............................................................................................................vii
List of Tables............................................................................................................vii
List of Tables...............................................................................................................vii
1. Abstract.....................................................................................................................1
1.1 Proposed Solution...................................................................................................1
2. Performance Testing Tools Evaluated.........................................................................2
2.1 HP LoadRunner and Performance Center....................................................................2
2.2 IBM Rational Performance Tester..............................................................................3
2.3 Borland Silk Performer............................................................................................3
2.4 Telerik Test Studio..................................................................................................3
2.5 Apache JMeter.......................................................................................................3
2.6 j-Chav..................................................................................................................4
2.7 Microsoft WCAT – Web Capacity Analysis Tool (formerly WAS – Web Application Stress
Tool)..........................................................................................................................4
2.8 Sun VisualVM.........................................................................................................4
2.9 WAPT...................................................................................................................5
2.10 Web Application Load Simulator (LoadSim)...............................................................5
2.11 Neo Load.............................................................................................................5
2.12 EggPlant Load Tester.............................................................................................5
2.13 OpenSTA – Open System Testing Architecture .........................................................6
2.14 OpenWebLoad......................................................................................................6
2.15 OpenLoad............................................................................................................6
2.16 loadUI.................................................................................................................6
2.17 LoadeaTest..........................................................................................................6
2.18 LoadManager.......................................................................................................7
2.19 Test Automation FX...............................................................................................7
2.20 CodedUI..............................................................................................................7
2.21 Soap UI...............................................................................................................7
2.22 Soap-Stone.........................................................................................................7
2.23 Web Performance Suite.........................................................................................8
2.24 Webserver Stress Tool...........................................................................................8
2.25 WebLOAD............................................................................................................8
2.26 QALoad...............................................................................................................8
2.27 Lisa Load.............................................................................................................8
2.28 Grinder...............................................................................................................9
2.29 nGrinder..............................................................................................................9
Copyrights © NetSol Technologies Ltd. All rights reserved iii
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Test Plan – Ascent –Version 0.1
2.30 GrinderStone.......................................................................................................9
2.31 MStone...............................................................................................................9
2.32 ValGrind..............................................................................................................9
2.33 QEngine..............................................................................................................9
2.34 Qtest................................................................................................................10
2.35 HammerHead 2..................................................................................................10
2.36 Hammerora........................................................................................................10
2.37 Pylot.................................................................................................................10
2.38 OpenWebLoad....................................................................................................10
2.39 SD’s Profiler.......................................................................................................11
2.40 Benerator..........................................................................................................11
2.41 SOATest (by Parasoft).........................................................................................11
2.42 Scapa Test and Performance Platform ...................................................................11
2.43 CLIF Load Injection ............................................................................................12
2.44 ContiPerf...........................................................................................................12
2.45 Coveriety...........................................................................................................12
2.46 D-ITG (Distributed Internet Traffic Generator).........................................................12
2.47 Database Opensource Test Suit (DOTS).................................................................13
2.48 DB Monster........................................................................................................13
2.49 Deluge..............................................................................................................13
2.50 DieselTest..........................................................................................................13
2.51 Faban................................................................................................................13
2.52 FunkLoad...........................................................................................................14
2.53 Curl-Loader........................................................................................................14
2.54 Spirant Avalanche...............................................................................................14
2.55 IXIA IxLoad........................................................................................................14
2.56 Paessler.............................................................................................................14
2.57 AQ TestComplete and AQ Time.............................................................................14
2.58 Test Maker.........................................................................................................15
2.59 TP Test..............................................................................................................15
2.60 Tsung................................................................................................................15
2.61 Tnsgen..............................................................................................................15
2.62 N-Time..............................................................................................................15
2.63 p-unit................................................................................................................16
2.64 Lobo.................................................................................................................16
2.65 MessAdmin........................................................................................................16
2.66 Multi Router Traffic Grapher (MRTR)......................................................................16
2.67 RRRD Tool.........................................................................................................16
2.68 Multi-Mechanize..................................................................................................16
2.69 Stress Driver......................................................................................................17
2.70 Load Impact.......................................................................................................17
2.71 Pure Load..........................................................................................................17
2.72 Pure Agent.........................................................................................................17
2.73 FWPTT...............................................................................................................17
2.74 JBlits Pro...........................................................................................................18
2.75 JCrawler............................................................................................................18
Copyrights © NetSol Technologies Ltd. All rights reserved iv
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Test Plan – Ascent –Version 0.1
2.76 PandoraFMS.......................................................................................................18
2.77 Siege................................................................................................................18
2.78 HTTP Load.........................................................................................................18
2.79 httPerf...............................................................................................................19
2.80 iPerf..................................................................................................................19
2.81 Site Blaster........................................................................................................19
2.82 RawLoadTester...................................................................................................19
2.83 Allmon..............................................................................................................20
2.84 AppsWatch.........................................................................................................20
2.85 BurnIn Test........................................................................................................20
2.86 Capacity Calibration (CapCal)...............................................................................20
2.87 Chariot..............................................................................................................21
2.88 CitraTestVU........................................................................................................21
2.89 SSW Performance Pro..........................................................................................21
2.90 e-Load..............................................................................................................21
2.91 MLoad...............................................................................................................21
2.92 Monitor Master...................................................................................................22
2.93 Forecast............................................................................................................22
2.94 GS DataGenerator...............................................................................................22
2.95 vPerformer.........................................................................................................22
2.96 JUnit.................................................................................................................22
2.97 Xceptance LoadTest............................................................................................23
2.98 zapwireless........................................................................................................23
2.99 Seagull..............................................................................................................23
2.100 SIPp................................................................................................................23
2.101 SLAMD............................................................................................................23
2.102 IxoraRMS.........................................................................................................24
2.103 j-hawk.............................................................................................................24
2.104 Wireshark........................................................................................................24
2.105 Ostinato...........................................................................................................24
2.106 Eclipse Test & Performance Tools Platform Project (EPTP)........................................24
2.107 Web Polygraph.................................................................................................24
3. Tools Selection Criteria.............................................................................................25
3.1 Business and Technical Requirements......................................................................25
3.2 Performance Test Objectives..................................................................................25
3.3 Performance Test Execution...................................................................................28
3.4 Load Description...................................................................................................28
4. Conclusion................................................................................................................29
4.1 Selected Tools......................................................................................................30
4.2 SWOT Analysis.....................................................................................................30
4.3 Training Needs.....................................................................................................31
4.4 Lessons Learnt.....................................................................................................31
5. Definition of Terms, Acronyms And Abbreviations.....................................................31
Copyrights © NetSol Technologies Ltd. All rights reserved v
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Test Plan – Ascent –Version 0.1
Document & Data Control
Document InformationDocument Information
Category Information
<Title> Performance Test Tool Selection
<Version> 0.1
<Author> Kamran Shaukat Ali Khan
<Identifier>
<Life Cycle Activity> TLC
<Issue Date> Pending
Document Review InformationDocument Review Information
Review Date Reviewer Name Version Reference / Evidence
Document Revision HistoryDocument Revision History
Author Date Version Description
Kamran
Shaukat Ali
Khan
15 May 2014 0.1 Draft
Copyrights © NetSol Technologies Ltd. All rights reserved vi
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Test Plan – Ascent –Version 0.1
List of FiguresList of Figures
Figure 1, Performance Acceptance at a glance..............................................................27
List of TablesList of Tables
Table 1. Tools fostering Performance Testing phases.....................................................2
Table 2. Selected tools for Performance Testing phases................................................30
Copyrights © NetSol Technologies Ltd. All rights reserved vii
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
1. Abstract
OUR PRODUCT is engineered for performance and agility.
Performance testing is the process of assessing performance/efficiency of a transaction system
whilst simultaneous users are accessing it; this is to determine if subject application (with all factors
involved) can serve its purpose efficiently. Performance Testing is a fundamental aspect of
software engineering, but it is a practice that too often is easily forgotten in today’s fast-paced
application development culture. Performance testing applications efficiently is essentially an
integral component for meeting end customer expectations.
The aim of this document is to evaluate and compare all available performance testing tools to
determine their usability and effectiveness. Software testing is a crucial part of software
development and the process of automating software testing is vital to its success. OUR PRODUCT
is a complex system and performance testing throughout the software development cycle is
pertinent to the success of the software.
To enable comparison of the automated performance testing tools, a multi-partitioned metric suite
was developed to guide the tool evaluation process. Inferences, implications and results are
presented and discussed in Section 3.
1.1 Proposed Solution
The overall goal of this research is to evaluate and compare several automated performance
testing tools to determine their usability and effectiveness for OUR PRODUCT. To accomplish this
goal, the research:
1. Prepared a comprehensive performance testing tools available on internet.
2. Broadened the scope of tools to include shareware, freeware, Open licenses and
open source tools. The list of tools, along with tools’ assessment is described in Section
2.
3. Defined criteria for tools selection based on business and technical requirements of
OUR PRODUCT; this will help narrow down the scope of research.
4. Compared existing tools in practice at ChromeIS with evolving requirements as
highlighted in above point.
5. Performed assessment of tools based on features and later, experiment with actual
application i.e. OUR PRODUCT (for e.g. CMS)
6. Tested the target applications using the selected automated testing tools and gather
resulting data
7. Evaluated and interpret results to draw inferences and make recommendations
OUR PRODUCT is a smart client being developed in Visual Studio 2010 using.net framework 4.5 – this imposes
certain limitations. An RND activity has been concluded where a number of tools were evaluated. The
assessment concluded that Performance Testing has to be carried out in Visual Studio .net itself. However,
there are certain other tools which can be utilized to foster in Performance Testing.
Below table enlists tools which will be used for Performance Testing as well as for resources monitoring:
Copyrights © NetSol Technologies Ltd. All rights reserved 1
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
Tool Purpose
Visual Studio 2013 For Performance Testing of areas which can’t be worked out
with HP PC / LR.
HP LoadRunner 12 Automated performance and stress testing
NeoLoad Trace client/server communication
SolarWind Defects logging and tracking
Spotlight Enterprise Resource Monitoring
Windows Profiler Extensive Resource Monitoring
Table 1. Tools fostering Performance Testing phases
2. Performance Testing Tools Evaluated
The tools summarized in this section constitute a concise list of tools available over internet for
automated performing testing (over a hundred tools were looked into). Each tool was evaluated
based on the criteria defined in Section 3.
2.1 HP LoadRunner and Performance Center
HP LoadRunner is a cost-effective and comprehensive tool used for automated performance
testing. This is our first choice because:
• LR is industry leader with over 90% market share
• ChromeIS has most experience in HP LR/PC
• Fully trained resources available
• Uses standard programming techniques and common languages such as C, VB and
Java; thus eliminating need to learn new language.
• Easy to script, parameterize, correlate and close to real test execution
• Extensive reporting, graphs, statistics collection, helping in root cause identification
HP LoadRunner brought support for .net protocol in version 12 which was released last month (15 th
April 2014); however, the support (as in readme file) does not describe component level details
which can be recognized by LoadRunner.
Through experiments, we can conclude that LR does not recognize OUR PRODUCT CMS client. It
fails with the most basic level of scripting (login only – without selecting a branch) and the
LoadRunner component i.e. VUGen stops responding. The script generated fails to compile and
introduces number of errors.
We can keep an eye on new patches of LR and keep experiments alive. However, this must not
stop overall performance test progress.
As of now, the tool is not a match to OUR PRODUCT requirements and cannot facilitate at any
level or phase.
Copyrights © NetSol Technologies Ltd. All rights reserved 2
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
2.2 IBM Rational Performance Tester
IBM RPT was 2nd
choice since they hold nearly 6% out of 10% market share (after 90% being with LR).
IBM RPT is slow in scripting, intelligence, execution and other aspects. However, nowhere on their
website IBM claims that RPT supports .net or WCF or WPF.
A formal conversation was opened with IBM but there had been no response.
We can keep an eye on IBM newsletter to see if they provide any sophisticated mechanism for
performance testing of WCF or SOA based architecture.
As of now, the tool is not a match to OUR PRODUCT requirements and cannot facilitate at any
level or phase.
2.3 Borland Silk Performer
Silk Performer 15 now enables to capture all the raw measure data of a load test. It now
automatically detects asynchronous communication patterns such as polling and long-polling. It
more effectively supports applications with periodic, event-based, or server-triggered content
updates.
Tool is not as rich and introduced support for Ajax just recently with no support for modern needs of
RIA based clients and smart clients.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.4 Telerik Test Studio
The tool fails to recognize Telerik “ribbon” components. This was discussed with Telerik support and
they confirmed the problem exists and listed as known issues.
Since the tool fails to recognize the first component which loads after login, the tool cannot be
used.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.5 Apache JMeter
Through experiments, we can conclude the JMeter is able to script the calls, however, since OUR
PRODUCT uses encryption, JMeter fails to decrypt the client/server communication. This is
necessary since the JMeter cannot process an encrypted call.
Ideally, we require JMeter to decrypt the server response, process it as required, re-crypt the newly
formulated request and then send it to the server. JMeter needs to do it for each response from
server to simulate load.
Another workaround could be to remove encryption from OUR PRODUCT as long as application is
undergoing performance testing. Although this imposes possible concern like:
• The encryption and decryption is costly activity and requires additional layer of effort
• The performance measures may not be realistic (slightly improved than real) since
client and server don’t have to encrypt or decrypt
Copyrights © NetSol Technologies Ltd. All rights reserved 3
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
If the above constraint can be lived with, we can consider JMeter further.
2.6 j-Chav
JChav is a way to see the change in performance of a web application over time, by running a
benchmark test for each build. JChav reads all the JMeter logs from each run (one per build), and
produces a set of charts for each test in each run.
The tool compliments JMeter and hence cannot be right fit for OUR PRODUCT.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.7 Microsoft WCAT – Web Capacity Analysis Tool
(formerly WAS – Web Application Stress Tool)
WCAT (Web Capacity Analysis Tool) is an excellent tiny tool from Microsoft to perform load test
web application on IIS. This tool enables users to do performance analysis on various scenarios of
web application. All “perfmon” performance counters (like processor time, private bytes usage,
disk queue length, total bytes sent or received in network) can be specified for the performance
testing.
Once scenario in application (i.e. appropriate URLs) is specified, a number of virtual clients from
various client machines will visit the URL with appropriate request data. Like other performance
tools, the following players are involved in a performance testing:
• Server – machines on which your web application components are running
• Client – A virtual client on a machine which acts as end-user to visit the URL
• Controller – This coordinates a test among the virtual clients on various machines. It also
captures and collates the performance counters from appropriate servers.
• Works in Visual FoxPro.
However, with all the excellence this tool may have, it is unable to recognize OUR PRODUCT
application components.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.8 Sun VisualVM
VisualVM is a free visual tool, originally from Sun, to monitor and troubleshoot Java applications.
Runs on Sun JDK 6, but is able to monitor applications running on JDK 1.4 and higher. Utilizes various
available technologies like jvmstat, JMX, the Serviceability Agent (SA), and the Attach API to get
data and uses minimal overhead on monitored applications. Capabilities include: automatically
detects and lists locally and remotely running Java applications; monitor application performance
and memory consumption; profile application performance or analyze memory allocation; is able
to save application configuration and runtime environment together with all taken thread dumps,
heap dumps and profiler snaphots into a single application snapshot which can be later processed
offline.
Tool is more focused on java based applications. The tool does not address modern needs of RIA
based clients and smart clients.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
Copyrights © NetSol Technologies Ltd. All rights reserved 4
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
2.9 WAPT
WAPT can test and analyze the performance characteristics and bottlenecks of web under various
load conditions with dynamic content and secure HTTPS pages. Real User Simulation with Dynamic
Test Data, Recording and Playback of HTTPS/SSL Pages, Flexible Test Volume & Load Definition,
Clear Reports and Graphs, Full Log of Virtual Users Activities Adjustable delays ("think times")
between requests to simulate real user behavior Support of redirects, including redirects to another
server Support of all types of proxies.
The tool is very mature but it lacks support of WPF, WCF and smart clients.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.10 Web Application Load Simulator (LoadSim)
LoadSim is a web application load simulator. It allows you to create simulations and have those
simulations run against your webserver.
It’s a web application load simulator. It allows you to create simulations and have those simulations
run against your webserver.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.11 Neo Load
NeoLoad is a professional load testing software for web applications (ASP.Net, J2EE, PHP, etc.)
Works almost without scripting, automatic handling of ASP.Net VIEWSTATE, Automatic support of
user sessions based on cookies or URL rewriting, Realistic VU simulation, cookies handling, plays
simultaneously Virtual Users with different behaviors, modem emulation, IP spoofing, SSL support
(record and playback), HTML/PDF/XML reporting, Compare results of several tests, Controller and
Load Injectors are multi-platform: Windows, Unix.
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well. However, it successfully transcribes the sequence and composition of
calls made to web services. This can be used in case the composition is not known to performance
tester.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.12 EggPlant Load Tester
eggPlant performance enabled HTTP-oriented Java and .NET Web Virtual Users (VUs) to be easily
generated by recording use-cases.
eggPlant Performance is designed for asynchronous requests, push technology, HTTP streaming,
and dynamic data originated by the client by supporting multiple threads of execution and
intelligent script generation.
Some applications, however, simply cannot be automated at the HTTP level and are best
automated at the GUI application level. In these situations eggPlant Performance can be used
with eggPlant Functional or Selenium VUs for application level testing.
eggPlant has in-built support for driving common non-web client-side APIs such as .NET and Java.
This leverages the native support for security, session management, and data serialization making
test creation faster, more effective, and more robust. The tool supports mobile clients, complex
thick clients and propriety devices. More information is available.
Copyrights © NetSol Technologies Ltd. All rights reserved 5
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
The tool needs further RND to reach a conclusion.
2.13 OpenSTA – Open System Testing Architecture
OpenSTA is a distributed software testing architecture based on CORBA. Using OpenSTA (Open
System Testing Architecture) a user can generate realistic heavy loads simulating the activity of
multiple concurrent virtual users. OpenSTA graphs both virtual user response times and resource
utilization information from all Web Servers, Application Servers, Database Servers and Operating
Platforms within environment, so that precise performance measurements can be gathered during
load tests and analysis on these measurements can be performed.
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.14 OpenWebLoad
OpenWebLoad is a tool for load testing web applications. It aims to be easy to use and providing
near real-time performance measurements of the application under test.
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.15 OpenLoad
OpenLoad provides no mechanism to test desktop based applications – this in turns disregards
smart client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.16 loadUI
loadUI tool focused mainly on Light Directory Access Protocol (LDAP) and web applications
working on HTTP, Web Services, REST, AMF, JMS, JDBC as well. The tool appears to be limited in
functionalities it offers. The tool is incapable of handling Rich Internet Application (RIA) such as flex
or Silverlight.
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
2.17 LoadeaTest
LoadeaTest works in C# and .net and has various components. Capture module creates new test,
modelize it as a script, enhances, replays and validates it. Control module defines test properties to
launch test on several machines, monitors test in real time and gives a results view. An injection
module executes script and collect data on different machines. Collect module is an http
interface between all modules and database for storage dump. Last, analysis module is to view
test results as tables and graphs, to generate reports, and to activate statistic algorithm on data
(Principle Component Analysis).
It works with HTTP, TCP, SOAP, RSS System resources. Scripts in high level language (C#, VB.NET )
and data in XML format.
Copyrights © NetSol Technologies Ltd. All rights reserved 6
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
The tool provides no mechanism for WCF specific applications in .net.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.18 LoadManager
LoadManager was developed in Java (Eclipse platform). The result is a testing tool which runs on
all platforms supported by Eclipse and Java such as Linux, Windows, HP Unix, etc. Supported
protocols are: HTTP, HTTPS, Socket (TCP/IP), JMS, RMI, scripts in high level language (C#, VB.NET )
and data in XML format. Color-coding shows which users pass/fail and a script-link is provided*
Testing of individual or groups of users can be paused/restarted at will.
The tool provides no mechanism for WCF specific applications in .net.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.19 Test Automation FX
TBC
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.20 CodedUI
TBC
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.21 Soap UI
TBC
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.22 Soap-Stone
Soap Stone is a Network benchmark application which can put your network under load and
conduct automatic benchmark and recording activities
The tool is limited to network load. This is very basic to our requirement.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
Copyrights © NetSol Technologies Ltd. All rights reserved 7
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
2.23 Web Performance Suite
Web Performance Suite includes a client-side analysis module (Analyzer) a load testing module
(Load Tester) and server analysis modules. It is capable of handling most popular web-design
technologies such as AJAX, Web forms, J2EE, ASP, .NET/C#, PHP, ColdFusion, Java, etc.
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.24 Webserver Stress Tool
Webserver Stress Tool is a HTTP-client/server test application
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.25 WebLOAD
WebLOAD is an Open Source, available for free download and use, WebLOAD is a commercial-
grade open source project with more than 250 engineering years of product development.
Companies that require commercial support, additional productivity features and compatibility
with third-party protocols have the option of purchasing WebLOAD Professional.
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.26 QALoad
QALoad is a performance testing tool from Micro Focus used to perform repeatable load testing
and determine the ultimate performance and potential limits of a system
QALoad has its own language, EasyScript. It has no support for Silverlight or Smart Client.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.27 Lisa Load
LISA offers direct functional and performance validation of every component of distributed
applications, with or without a UI present. LISA load tests also leverage the end-to-end functional
tests created in LISA, invoking and verifying at every tier of the architecture, with a very low system
footprint required, so capacity can be simulated at the lowest possible effort and cost.
Lisa Load supports only Java/Swing; hence making it as not a choice for OUR PRODUCT.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
Copyrights © NetSol Technologies Ltd. All rights reserved 8
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
2.28 Grinder
Grinder tool focused mainly on Light Directory Access Protocol (LDAP) and web applications
working on HTTP protocol. The tool appears to be limited in functionalities it offers. The tool is
incapable of handling Rich Internet Application (RIA) such as flex or Silverlight.
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.29 nGrinder
nGrinder is a stress testing tool, executing script creation, test execution, monitoring, and result
report generator simultaneously.
Since Grinder itself is not applicable to OUR PRODUCT, the GrinderStone is not applicable either.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.30 GrinderStone
GrinderStone is an Eclipse plug-in for Grinder load testing scripts development including
debugging, modularity and pretty logging.
Since Grinder itself is not applicable to OUR PRODUCT, the GrinderStone is not applicable either.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.31 MStone
Mstone started as a mail performance measurement system but now can simultaneously test SMTP,
POP, IMAP, and some HTML based systems. It measures transaction latency in multiple stages, and
graphs the combined results from multiple clients.
MStone provider no mechanism for RIA or Smart clients.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.32 ValGrind
Valgrind is focused on Linux programs. Built in tools automatically detect many memory
management and threading bugs. ValGrind is not robust enough for RIA and smart clients.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.33 QEngine
AdventNet QEngine WebTest provides accurate way to test the functionality of Web applications
in both Windows and Linux machines. It provides an easy-to-learn user interface to completely
automate the creation of sophisticated test scripts, allows record and playback Web browser
events in IE and Mozilla browsers.
The tool is immature and focused more on Regression automation with no support for Silverlight.
Copyrights © NetSol Technologies Ltd. All rights reserved 9
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
It is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.34 Qtest
Qtest is automated load testing and monitoring solution, offering high capacity load simulation,
extensive monitoring capability. Qtest supports all Web, Web Service, J2EE, .Net, ASP, AJAX, CGI,
and Mainframe Portal environments and, in particular, complex Web applications like Siebel, SAP,
and Epiphany. With its non-protocol based Winload module, Qtest also supports any non-web
application.
The tool offers sophisticated and complex frameworks, but nowhere does it provide Silverlight or
Smart Client based on WCF.
It is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.35 HammerHead 2
Hammerhead 2 is a stress testing tool designed to test out your web server by initiating multiple
connections from IP aliases and simulated numerous (256+) users at any given time. The rate at
which Hammerhead 2 attempts to pound web server site is fully configurable.
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.36 Hammerora
Hammerora is a load generation tool for the Oracle Database and Web Applications. Hammerora
includes pre-built schema creation and load tests based on the industry standard TPC-C and TPC-H
benchmarks to deploy against the Oracle database with multiple users. Hammerora also converts
and replays Oracle trace files and enables Web-tier testing to build bespoke load tests for your
entire Oracle application environment.
The tool provides no mechanism to test web services or RIA based or desktop based smart client
applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.37 Pylot
Pylot is open performance and scalability testing tool, capable of running HTTP load tests to
compliment capacity planning, benchmarking, analysis, and system tuning. Pylot generates
concurrent load (HTTP Requests), verifies server responses, and produces reports with metrics. Tests
suites are executed and monitored from a GUI.
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.38 OpenWebLoad
preVue-X provides non-intrusive regression and performance testing of X-Window applications.
preVue-X accurately emulates multiple users while requiring no special hooks into your application
Copyrights © NetSol Technologies Ltd. All rights reserved 10
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
or X Libraries, thus guaranteeing that the application tested is the same application that will be
deployed. preVue-X operates at the X protocol level and independently of the GUI. It
automatically collects all user interactions with the application (e.g., mouse movements, button
activity, keystrokes, think time delays) into an easily editable test script for playback. Features
include automated synchronization and time-stamping; geometry independence and script
viewer
The tool does not speak about WCF in specific or smart clients. Need further RND to reach a
conclusion.
2.39 SD’s Profiler
SD's Profiler collects information about execution, frequency of all blocks of code in an application.
A graphical display sorts this information by file or by frequency, and the relative execution costs
are shown graphically superimposed over the source code. Available for C, COBOL, and Java. For
C and Java, easily customized to work in embedded or real-time contexts.
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.40 Benerator
benerator is a framework for creating realistic and valid high-volume test data, used for
(unit/integration/load) testing. Metadata constraints are imported from systems and/or
configuration files. Data can be imported from and exported to files and systems, anonymized or
generated from scratch. Domain packages provide reusable generators for creating domain-
specific data as names and addresses internationalizable in language and region. It is
customizable with plugins and configuration options.
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.41 SOATest (by Parasoft)
Parasoft SOA Quality Solution continuously measure how each service conforms to the often
dynamic expectations defined by both your own organization and your partners, reduces the
complexity of testing in today’s heterogeneous environment – with limited visibility/control of
distributed components or vendor-specific technologies. It validates all critical aspects of complex
transactions, which may extend beyond the message layer through a web interface, ESBs, etc.
The tool provides no clarity in order to test WCF or smart client or web services.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.42 Scapa Test and Performance Platform
Scapa Test and Performance Platform is testing, diagnosis and monitoring product. The Platform
works by simulating multiple users of a computer system. By measuring the performance seen by
virtual users, it is possible to understand how the system would behave under a corresponding level
of real usage.
Copyrights © NetSol Technologies Ltd. All rights reserved 11
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.43 CLIF Load Injection
CLIF is a modular and flexible distributed load testing platform. It may address target systems
reachable from a Java program (HTTP, DNS, TCP/IP...) CLIF provides 3 user interfaces (Swing or
Eclipse GUI, command line) to deploy, control and monitor a set of distributed load injectors and
resource consumption probes (CPU, memory...) An Eclipse wizard helps programming support for
new protocols. Load scenarios are defined through XML-editing, using a GUI, or using a capture
tool. The scenario execution engine allows the execution of up to millions of virtual users per load
injector.
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.44 ContiPerf
ContiPerf is a small testing utility that enables the user to easily leverage JUnit 4 test cases as
performance tests e.g. for continuous performance testing. It is inspired by JUnit 4's easy test
configuration with annotations and by JUnitPerf's idea of wrapping Unit tests for performance
testing, but more powerful and easier to use.
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.45 Coveriety
TBC
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.46 D-ITG (Distributed Internet Traffic Generator)
D-ITG (Distributed Internet Traffic Generator) is a platform capable to produce traffic at packet
level accurately replicating appropriate stochastic processes for both IDT (Inter Departure Time)
and PS (Packet Size) random variables.
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
Copyrights © NetSol Technologies Ltd. All rights reserved 12
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
2.47 Database Opensource Test Suit (DOTS)
The Database Opensource Test Suite (DOTS) is a set of test cases designed for the purpose of stress-
testing database server systems in order to measure database server performance and reliability.
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.48 DB Monster
DBMonster is an application to generate random data for testing SQL database driven
applications under heavy load.
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.49 Deluge
Deluge is an open-source web site stress test tool. It simulates multiple user types and counts,
includes proxy server for recording playback scripts, and log evaluator for generating result
statistics.
This tool has not been updated since 2002. This tool is no longer under active development
although it is still available on Sourceforge.
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.50 DieselTest
Dieseltest is a Windows application that simulates hundreds or thousands of users hitting a website.
BEWARE: This tool has not been updated since 2001. It remains listed here in case anybody wishes
to take it over.
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.51 Faban
Faban is a facility for developing and running benchmarks, developed by Sun. It has two major
components, the Faban harness and the Faban driver framework. The Faban harness is a harness
to automate running of server benchmarks as well as a container to host benchmarks allowing
new benchmarks to be deployed in a rapid manner. Faban provides a web interface to launch &
queue runs, and extensive functionality to view, compare and extract graph of outputs.
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
Copyrights © NetSol Technologies Ltd. All rights reserved 13
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.52 FunkLoad
FunkLoad is a functional and web performance testing tool, written in Python. It can be used for
functional and regression testing of web projects as well as performance testing and stress testing
to overwhelm the web application resources and test the application recoverability.
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.53 Curl-Loader
Written in C, Curl-Loader is a web application testing and load generating tool. Provides an open-
source alternative to Spirent Avalanche and IXIA IxLoad. The loader uses real HTTP, FTP and TLS/SSL
protocol stacks, simulating tens of thousand and hundred users/clients each with own IP-address.
The tool supports user authentication, login and a range of statistics.
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.54 Spirant Avalanche
TBC.
2.55 IXIA IxLoad
TBC.
2.56 Paessler
TBC
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.57 AQ TestComplete and AQ Time
AQTime is a performance profiler and memory allocation debugger for 32-bit and 64-bit Windows
and .NET-connected applications. AQtime can be integrated into Microsoft Visual Studio,
Borland Developer Studio or CodeGear RAD Studio or CodeGear RAD Studio or it can run as a
stand-alone application. AQtime can also determine what caused bottlenecks.
The tool provides no mechanism to test RIA based applications or WCF calls.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
Copyrights © NetSol Technologies Ltd. All rights reserved 14
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
2.58 Test Maker
TestMaker offers building and running intelligent test agents that test Web-enabled applications for
scalability, functionality, and performance. It works with object-oriented scripting language
(Jython) to build intelligent test agents, an extensible library of protocol handlers (HTTP, HTTPS,
SOAP, XML-RPC, SMTP, POP3, IMAP), a new agent wizard featuring an Agent Recorder to write
scripts, a library of fully-functional sample test agents, and shell scripts to run test agents from the
command line and from unit test utilities.
The tool provides no mechanism to test RIA and smart client.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.59 TP Test
TPTEST measures the throughput speed to and from various reference servers on the Internet. The
use of TPTEST may help increase the consumer/end user knowledge of how Internet services work.
The tool is very basic and cannot be graded as performance testing tool for applications.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.60 Tsung
Tsung is a distributed load testing tool. It is protocol-independent and can currently be used to
stress HTTP, SOAP and Jabber servers (SSL is supported). It simulates complex user's behavior using
an XML description file, reports many measurements in real time (including response times, CPU
and memory usage from servers, customized transactions, etc.). HTML reports (with graphics) can
be generated during the load. For HTTP, it supports 1.0 and 1.1, has a proxy mode to record
sessions, supports GET and POST methods, Cookies, and Basic WWW-authentication. It has already
been used to simulate thousands of virtual users.
The tool requires further RND to experiment with OUR PRODUCT POS.
2.61 Tnsgen
Tnsgen is test data generation tool which operates under Solaris, Linux and Windows, capable of
generating thousands of records per second in XML, SQL or user-defined formats. Easily extensible
to include project-specific data sets.
The tool is a data generating tool, it cannot be graded as performance testing tool.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.62 N-Time
N-Time tool is very similar to NUnit tool to perform repeatable tasks that help managers, architects,
developers and testers to test an application against its performance.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
Copyrights © NetSol Technologies Ltd. All rights reserved 15
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
2.63 p-unit
p-unit is an open source framework for unit test and performance benchmark, under GPL license.
p-unit supports to run the same tests with single thread or multi-threads, tracks memory and time
consumption, and generates the result in the form of plain text, image or pdf file.
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.64 Lobo
Lobo tool supports performance testing of Java based applications only.
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.65 MessAdmin
MessAdmin is a light-weight and non-intrusive notification system and HttpSession administration for
J2EE Web Applications, giving detailed statistics and informations on the application. It installs as a
plug-in to any Java EE WebApp, and requires zero-code modification.
The tool is specific for java based application. It provides no mechanism to test desktop based
applications – this in turns disregards smart client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.66 Multi Router Traffic Grapher (MRTR)
Multi Router Traffic Grapher is written in perl and works on Unix/Linux as well as Windows and even
Netware systems. MRTG is free software licensed under the Gnu GPL.
The tool is specific for java based application. It provides no mechanism to test desktop based
applications – this in turns disregards smart client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.67 RRRD Tool
RRRD Tool is data logging and graphing system for time series data. It can be used to write custom
monitoring shell scripts or create whole applications using its Perl, Python, Ruby, TCL or PHP
bindings.
The tool is barely into application performance testing.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.68 Multi-Mechanize
Multi-Mechanize is an open source framework for performance testing. It allows running
simultaneous python scripts to generate load against a web site or web service.
Copyrights © NetSol Technologies Ltd. All rights reserved 16
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
The tool does not address modern needs of RIA based clients and smart clients.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.69 Stress Driver
Stress Driver is a general-purpose stress test tool.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.70 Load Impact
LoadImpact is online load testing service from loadimpact.com for load- and stress- testing; with
access to distributed network of load generator nodes – server clusters with very fast connections
to enable simulation of simultaneous users accessing your website concurrently. Free low level load
tests for 1-50 simulated users; higher levels have monthly fees.
No claim to test RIA client on their site. No support for smart client.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.71 Pure Load
PureLoad is a load testing tool that simulates concurrent users executing requests against server
based applications. Use PureLoad to verify performance and stability of your server applications.
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.72 Pure Agent
PureAgent is a performance monitoring tool that replicates real users generating activities on
server applications. PureAgent measures the total round trip time it takes to execute one or many
user operations accessing either static or dynamic content. The results are presented as charts in a
standard web browser, alarm levels can be defined which might trigger notifications to be sent by
Email. The monitored scenario of user operations is created using the PureTest application.
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.73 FWPTT
fwptt is used for load testing of web applications only. It can record normal and ajax requests. It
can understand HTML being returned from server, hence, it works with asp.net, jsp, php or other
such applications. The tool recently supported AJAX as well.
Since AJAX is the extent of the tool, the tool is too basic to meet the complex requirements of OUR
PRODUCT where web is using silverlight and smart client.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
Copyrights © NetSol Technologies Ltd. All rights reserved 17
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
2.74 JBlits Pro
JBlits gives continuous heavy loading and automated response validation. Easy to use, quick to
setup. Fully featured with logging, graphs, stats, scheduling, HTTP debugger etc. Full Java API.
Works in Java, works for Java. The tool is not meant for Silverlight and smart client.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.75 JCrawler
JCrawler can start with a set of starting URLs and it will begin crawling from that point onwards,
going through any URLs it can find on its way and generating load on the web application.
Although, the load parameters (hits/sec) are configurable, JCrawler doesn’t look like a real
performance testing tool. There are no built-in mechanism for correlation and parameterization.
The tool, at best, can simulate load on static websites where no custom data is required.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.76 PandoraFMS
Pandora FMS is a monitoring Open Source software and watches systems and applications.
Pandora FMS could detect a network interface down, defacement in website, a memory leak in
server application, or the movement of any value of the NASDAQ new technology market.
Pandora FMS could send out SMS message when system fails... or when Google's value drop below
US$ 500.
Although PandoraFMS provides features unique to itself and it can be very useful in many ways,
PandoraFMS fails to simulate load on RIA and Smart Client.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.77 Siege
Siege can be used to test over HTTP and HTTPS. SIEGE is an http regression testing and
benchmarking utility, designed to measure the performance of code. It lets the user hit a
webserver with a configurable number of concurrent simulated users. SCOUT surveys a webserver
and prepares the urls.txt file for a siege. In order to perform regression testing, siege loads URLs from
a file and runs through them sequentially or randomly. Scout makes the process of populating that
file easier. You should send out the scout, before you lay siege.
With the above limitation of the tool, Siege is not a match to OUR PRODUCT requirements and
cannot facilitate at any level or phase.
2.78 HTTP Load
http_load is a multiprocessing http test client. As the name suggests, the tool works specifically for
HTTP protocol. This helps cater plain HTML applications only.
http_load runs multiple http fetches in parallel, to test the throughput of a web server in a single
process. It can be configured to do https fetches as well. It processes a file containing a list of URLs
that may be fetched, a flag specifying how to start connections (either by rate or by number of
simulated users), and a flag specifying when to quit (either after a given number of fetches or a
Copyrights © NetSol Technologies Ltd. All rights reserved 18
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
given elapsed time). There are also optional flags for checksums, throttling, random jitter, and
progress reports.
With all above, the tool serves very basic level requirement for OUR PRODUCT. Provides no
mechanism to test RIA based and desktop based applications – this in turns disregards smart client
applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.79 httPerf
httperf provides facility for generating HTTP workloads. The three distinguishing characteristics of
httperf are:
• its robustness, which includes the ability to generate and sustain server overload,
• support for the HTTP/1.1 and SSL protocols, and
• its extensibility.
With all above, the tool serves very basic level requirement for OUR PRODUCT. Provides no
mechanism to test RIA based and desktop based applications – this in turns disregards smart client
applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.80 iPerf
iperf was developed as a modern alternative for measuring maximum TCP and UDP bandwidth
performance. iperf allows the tuning of various parameters and UDP characteristics. iperf reports
bandwidth, delay jitter, datagram loss.
With all above, the tool serves very basic level requirement for OUR PRODUCT. Provides no
mechanism to test RIA based and desktop based applications – this in turns disregards smart client
applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.81 Site Blaster
Site Blaster is a web site load and stress testing tool. It can be used to rapidly submit requests to a
site. Or, it can pause a random amount of time between submissions; approximating the behavior
of a user. While the testing is being performed, the pages being tested will be displayed. When
testing is complete, a report is available that can be viewed or printed.
It provides no mechanism to test RIA based and desktop based applications – this in turns
disregards smart client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.82 RawLoadTester
RLT calls the selected URL multiple times and tells server response time. It writes some additional
runtime details to the PHP log file to optionally do more granular analysis afterwards. Although the
server processes most of the statistics, all URL requests come from the browser. You can run as
many browsers and workstations simultaneously as you want.
Copyrights © NetSol Technologies Ltd. All rights reserved 19
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
RLT provides no mechanism to test RIA based and desktop based applications – this in turns
disregards smart client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.83 Allmon
Allmon is an open source, generic system collecting and storing metrics used for performance and
availability monitoring.
It provides no mechanism to test RIA based and desktop based applications – this in turns
disregards smart client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.84 AppsWatch
Appswatch is an automated testing tool for performance, and GUI testing and SLA monitoring and
alerting. It is protocol independent and works with any application accessible from windows
desktop. It works for Citrix, Java, .NET, Oracle, SAP, Eclipsys, Mckessen, Meditech, GE Healthcare,
Epic, Cerner, Client-Server, Flex, Flash and even custom applications
It provides no mechanism to test RIA based and desktop based applications – this in turns
disregards smart client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.85 BurnIn Test
BurnInTest allows sub-systems of a computer to be simultaneously stress tested for endurance,
reliability and stability. BurnInTest tests the CPU, hard drives, CD ROMs, DVDs, sound cards, 2D
graphics, 3D graphics, RAM, network connections & printers. The Pro version can also test tape
drives, USB, serial and parallel ports. This can all be done simultaneously as the application is
multithreaded.
It provides no mechanism to test RIA based and desktop based applications – this in turns
disregards smart client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.86 Capacity Calibration (CapCal)
CapCal writes code in Java for measuring web site performance under various loads--by actually
generating those loads, in real time, from computers scattered all across the Internet. CapCal can
also be distributed across your LAN and WAN, giving realistic performance data that matches end
user's experience.
It provides no mechanism to test RIA based and desktop based applications – this in turns
disregards smart client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
Copyrights © NetSol Technologies Ltd. All rights reserved 20
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
2.87 Chariot
Chariot tests and troubleshoots any segment of network and provides comprehensive reports of
the results by emulating transaction traffic from real applications. It can be used to predict how
well network infrastructure will handle popular applications prior to deployment, and to isolate and
evaluate the performance of new network devices. Chariot is highly scalable and can be used to
set up and control tests—emulating up to 10,000 connections and hundreds of thousands of end-
users—all from a central console.
It cannot be graded as real performance testing tool since it aims for network and infrastructure
against internet applications.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.88 CitraTestVU
CitraTestVU is a load testing tools, works in Visual Basic scripting to simulate Load on Citrix &
Microsoft Terminal Services Hosted Environments.
The tool is focused on Java only. It provides no mechanism to test RIA based and desktop based
applications – this in turns disregards smart client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.89 SSW Performance Pro
SSW Performance PRO! isolates performance bottle-necks in your Access 97, 2000, 2002 or 2003
app. It checks every form, combo and list box, and reviews SQL statements. It reports exactly what
to fix to improve the speed of your application. It’s especially useful after you've upsized an
application to SQL Server to pinpoint black spots in your app.
The tool is very basic and mainly targets MS Access. It cannot be graded a commercial level tool.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.90 e-Load
e-Load is enables accurate testing of the response times and scalability of web applications and
web services. e-Load can concurrent users, executing real business transactions to analyze how
well web software applications will perform under load.
The tool provides no mechanism to test RIA based and desktop based applications. It supports
Web Services but not WCF.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.91 MLoad
MLoad generates high levels of realistic network and user traffic ensuring that WAP Gateways,
MMSCs and web based mobile infrastructure elements meet the high-bandwidth, high-
performance demands of a mobile internet service. The toolset includes a test harness to allow
acceptance testing of mobile elements against performance requirements in the deployed
environment.
The tool focused on data flux and network bandwidth only. It provides no mechanism to test RIA
based and desktop based applications. It supports Web Services but not WCF.
Copyrights © NetSol Technologies Ltd. All rights reserved 21
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.92 Monitor Master
Monitor Master is a quality management solution for wireless data services - WML/XHTML, SMS and
MMS. Monitor Master helps Operators and Content Developers ensure that end-users of wireless
applications will have the best quality of experience possible - regardless of the handset being
used.
The tool focuses wireless protocols and not RIA based and desktop based applications. It supports
Web Services but not WCF.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.93 Forecast
Forecast is a easily and accurately tests the performance and scalability of IT applications and
systems.
No support for WPF, WCF or Smart Client.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.94 GS DataGenerator
GS DataGenerator generates high volumes of 'intelligent' test data on virtually any relational
database, creates business trends and patterns, simulates business workflow, mimics potential data
errors Employ number of predefined functions. This broad range of functionality allows testing
database performance and functionality during development.
It’s very basic tool, works in DOS and provides no support for WPF, WCF or Smart Client.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.95 vPerformer
vPerformer is used to assess the performance and scalability of web applications. vPerformer
allows assesses how web application responds when it is concurrently accessed by a large number
of users. It can measure the performance characteristics of application by recording and replaying
automated scripts that simulate a large number of concurrent virtual users.
The tool is very basic and doesn’t address modern needs like Silverlight etc.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.96 JUnit
JUnitPerf is a collection of JUnit test decorators used to measure the performance and scalability of
functionality contained within existing JUnit tests
The tool is specific for Java related apps.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
Copyrights © NetSol Technologies Ltd. All rights reserved 22
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
2.97 Xceptance LoadTest
Xceptance LoadTest for web and Java and other apps includes recording capabilities. XLT Cloud
Service available. Tests implemented as JUnit 4 test cases. For web-based tests, the framework
provides a (headless) browser that can emulate Internet Explorer or Firefox behavior. Can execute
client-side JavaScript in the emulated web browsers and that way it simplifies the creation of test
cases for Web 2.0 applications. It is platform independent since the tool is implemented in Java;
test scripting in Java or Ruby. The tool is free for up to five virtual users.
The tool is focused on Java only. It provides no mechanism to test RIA based and desktop based
applications – this in turns disregards smart client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.98 zapwireless
zapwireless is a wireless throughput test tool which can be used to fully characterize the statistical
performance of a wireless link.
It provides no mechanism to test RIA based and desktop based applications – this in turns
disregards smart client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.99 Seagull
Seagull was primary aimed at IMS protocols, now it is a traffic generator for functional, load,
endurance, stress and performance tests for almost any kind of protocol. Currently supports
Diameter, XCAP over HTTP, TCAP (GSM Camel, MAP, Win) protocols.
It provides no mechanism to test RIA based and desktop based applications – this in turns
disregards smart client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.100 SIPp
SIPp is a performance testing tool for the SIP protocol. Its main features are basic SIPStone
scenarios, TCP/UDP transport, customizable (xml based) scenarios, dynamic adjustement of call-
rate and a comprehensive set of real-time statistics. It can also generate media (RTP) traffic for
audio and video calls.
Although Sipp is a rich tool and may be useful for specific purposes, it provides no mechanism to
test RIA based and desktop based applications – this in turns disregards smart client applications as
well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.101 SLAMD
SLAMD Distributed Load Generation Engine is a Java-based application designed for stress testing
and performance analysis of network-based applications.
SLAMD does not provide any mechanism to test RIA based and desktop based applications – this
in turns disregards smart client applications as well.
Copyrights © NetSol Technologies Ltd. All rights reserved 23
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.102 IxoraRMS
IxoraRMS is a monitoring tool with great visualization and customization capabilities. But it severely
lacks capabilities to support RIA and modern complex architecture.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.103 j-hawk
j-hawk is a Java based open source framework which can be incorporated within an application
for performance testing. The idea requires to define *module and its method inside the application
and register the same with j-Hawk. j-Hawk executes the modules and generates a graphical
performance report which can be analyzed to find performance bottleneck of your application.
The tool is specific for Java based application which is not the case with OUR PRODUCT.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.104 Wireshark
TBC
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.105 Ostinato
Ostinato is an open-source, cross-platform packet/traffic generator and analyzer with a friendly
GUI. It aims to be "Wireshark in Reverse" and thus become complementary to Wireshark.The tool is
specific for Java based application which is not the case with OUR PRODUCT.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.106 Eclipse Test & Performance Tools Platform Project
(EPTP)
EPTP supplies frameworks and services that allow software developers to build unique test and
performance tools, both open source and commercial, that can be easily integrated with the
platform and with other tools.
Eclipse Test & Performance Tools Platform Project (TPTP) evidently provides no mechanism to test
desktop based applications – this in turns disregards smart client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.107 Web Polygraph
Web Polygraph is a benchmarking tool for caching proxies, origin server accelerators, L4/7
switches, content filters, and other Web intermediaries. This tool focused mainly on Light Directory
Access Protocol (LDAP) and web applications working on HTTP protocol. The tool appears to be
limited in functionalities it offers. The tool is incapable of handling Rich Internet Application (RIA)
such as flex or Silverlight.
Copyrights © NetSol Technologies Ltd. All rights reserved 24
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
3. Tools Selection Criteria
Criteria for selection of Automated Performance Testing tool was defined before proceeding with
research activities. The criteria broadly address Business and Technical Requirements and
Objectives sought from Performance Testing. Below is a highlight of foundation of criteria:
3.1 Business and Technical Requirements
1. It is essential for the application is able to pass at least 1 test case to perform certain business
process. Although, it is best to have reasonably stable application before performance testing
starts. If there is urgency, the application should be able to work under at least 1
situation/record-set.
2. It is pivotal to finalize performance test objectives before test scripting is begun. Number of
users, data volume and other variables must be known beforehand otherwise unnecessary
delays may occur. For example, if the intention is to run endurance or longevity test with 1000
users over 4.5M record base, for 24 hours; then test data requirement will be massive as
compared to running 100 users for 1 hour.
3. Dedicated test environment and interface systems are identified and requisition (booking)
made in advance and certified to be used for testing. The hardware specifications should be
documented for web server, application server, db server and any other layer involved. This
should be documented for Load Injectors/Generators as well.
4. System components have been identified and deployed over test environment mimicking (or
a close match to) production server.
5. Test IDs/passwords and access rights have been identified, requisition made and approval
granted for all involved applications. Test IDs must have the correct access rights for testing for
application as required by user’s role.
6. Criteria for test data to be used for testing has been drawn, and reviewed by respective
parties involved.
7. Test Data has been created in testing database.
8. System infrastructure and connectivity test are performed prior to each test phase to ensure
connectivity is enabled.
3.2 Performance Test Objectives
In general, the test objective is to ensure the respective applications meet the stated and implied
performance requirements of business; as well as, the statistics remain within the SLA agreed with
end customer.
Copyrights © NetSol Technologies Ltd. All rights reserved 25
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
1. Identify any bottlenecks and slow system performance
2. Ensure application response time is acceptable with data volume of 4.5 Million records
3. Gauge application responsiveness under Stress
4. Validate application behavior under endurance and longevity (24+ hours)
5. Validate the behavior of Load balancing over cluster nodes equally
6. To ensure that application is functioning properly under load
7. Establish performance benchmarks
8. Validate transactions response time against customers’ SLA
9. Perform root cause analysis for any problems and errors appearing under load – and
suggest possible optimization.
10. Draw projection as measures for future needs
11. Ensure smooth implementations with no/minimum disruption to business going live, and
applications rollout with minimum risk.
12. Project implementation compliance to ChromeIS processes.
13. Project quality that meets entry and exit criteria for each performance testing phase.
There are specific goals of performance testing, like:
1. Measure and benchmark response time for each page/screen to open
2. Measure and benchmark response time against each transaction, like, how long it
takes when, saving a proposal, saving a contract, converting a proposal into contract,
calculating exposure of a customer, asset is attached to a proposal, rentals are
generated, financial analysis is done, early termination is performed etc.
3. Measure and benchmark response time for each service as an individual.
4. This section specifies the test types (with definition for each) to run. This also explains
when application will be considered fit to undergo each type of testing and when a
particular
5. Measure and benchmark maximum number of users a transaction/service can
entertain disregarding response time (peak point identification)
6. Benchmark maximum number of users achieved against test environment
(hardware/software specifications)
7. Benchmark maximum hardware utilization against desired number of users (against
test environment specifications)
8. Benchmark system behavior against cold startup and warm startup
9. The spike ratio that the system must be capable of handling
10. The peak ratio that the system must be capable of handling?
11. Memory leakage against each business process or individual service
Copyrights © NetSol Technologies Ltd. All rights reserved 26
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
12. Identify which SPs fail and at what circumstances, at what line, under what conditions.
13. Facilitate in ensuring load is distributed evenly over all nodes of clusters (as configured)
14. Facilitate ensure seamless mitigation of failover under high load
15. Tolerance ratio: Imposed load ? 25 %?
16. Spike ratio: Imposed load x 3?
17. Burstiness ratio: Imposed load x 5?
18. Increase the load by multiplying the load baseline by 1x, 2x, 3x, 4x, Nx gradually until
unacceptable response time is reached through optimization.
19. Pass recommendation in achieving optimal configuration, or identifying target
component where optimization could be required.
0.1 Second
10 Second
20 Second
30 Second
Acceptable User Experience
Unacceptable User Experience
Questionable User Experience
Business is Closed
Response
Time
Load
60 Second
Figure 1, Performance Acceptance at a glance
The above assumes, all stakeholders are on same page regarding:
• What is transaction response time and what is acceptable response time?
• The difference between simultaneous and concurrent users
• Which metrics should be collected during all types of performance testing?
• What is the correlation between demand and increased load?
Copyrights © NetSol Technologies Ltd. All rights reserved 27
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
• How do we determine which components are problematic?
3.3 Performance Test Execution
Performance testing usually follows a linear path of events:
• Create performance-testing scripts and scenarios.
• Create Task Distribution Diagram to elaborate maximum number of users over a
component as well as maximum number of users accessing system at any point of
time.
• Execute performance tests as standalone tests to detect issues within a particular
business workflow. Repeat execution multiple times to surface most errors, identify root
cause of failure.
• Execute performance scenarios as a package (i.e. with all BPs together) to simulate
real life activities that are measured against performance success criteria.
• Monitor all variables involved in Performance Testing. This includes all servers and their
components such as processor, memory, NIC, clusters, caching, heap, top SQL
queries, SPs, ISS caching, IIS sessions, Kernel, Number of processes, Number of threads,
Throughput, DB Sessions, Parameters, Data Buffer Cache Hit Ratio, Shared Pool Hit
Ratio, Query to Parse ratio, PGA Usage, I/O read wait time, I/O write wait time, CPU
wait time, configurations, hits per second, timeouts etc. These statistics will be
collected for Web Server, App Server, DB Server and any other server involved, as well
as load injectors.
• Collate all statistics and observations. Perform analysis and document
recommendations/suggestions. Compile a concise performance testing results report.
• Tune the system.
• Repeat testing as required.
3.4 Load Description
In the context of Performance Testing, “Load” can be broadly classified into 2 categories, namely,
Server Based Load and User Based Load. Below is the elicitation of both terminologies.
Server-based Load
• Number of users and/or sessions
• Average session time
• Number of page views
• Average page views per session
• Peak period (e.g., 75% of traffic is from 11:00 AM-4:00 PM)
• Number of hits
• Average page size
• Most requested pages
• Average time spend on page
Copyrights © NetSol Technologies Ltd. All rights reserved 28
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
• Frequency of visits (e.g., 75% of users made one visit)
• Demographics
• Client information such as browser, browser version, Java script support, Java script
enable/disable, and so on.
User Based Load:
• Number of users
• Session length
• User activities and frequency of activities per session
• Think/Read/Data-input time
• Percentage by functional group
• Percentage by human speed
• Percentage by domain expertise/ familiarity (speed)
• Percentage by demographics (arrival rates)
Other questions to consider when describing system Load:
• What is the definition of “workload”?
• How do we size the workload?
• What is the expected workload?
• What’s the mix ratio of static pages vs. code?
• What is the definition of “increased load”?
• What is future growth? Can it be quantified?
• What is the definition of scalability?
A requirements document is filled up for each testing cycles to further clarify performance test
requirements. Refer to below example file.
4. Conclusion
After assessing large number of tools available in market, we can conclude that HP LoadRunner,
IBM Rational Performance Center and JMeter (the top 3 market leaders) do not directly support
testing the complex architecture we’re building; at least not as yet.
A number of open source tools, freeware utilities, ASM based performance testing services and
other freeware utilities do not address the performance testing challenges we‘re facing.
JMeter could be a possible option if encryption can be removed from OUR PRODUCT for testing
purposes.
EggPlant Load Tester described in detail how it can test web services written in .net. This requires
further RND and experimentation if time allows.
Tsung is an open source framework which claims testing web services and other complex niche
technology frameworks. This requires further RND and experimentation if time allows.
We’ve had trial runs with Visual Studio and optimistically, we can hope to meet our objectives with
MSVS. This is further described in Section 4.1 below.
Copyrights © NetSol Technologies Ltd. All rights reserved 29
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
4.1 Selected Tools
The testing tool chosen is Microsoft Visual Studio 2013.
NeoLoad can be used to identify the sequence of calls made to web services.
Although, MSVS this is not specifically designed performance testing in mind, however, this has
been picked as a workaround for current situation. Visual Studio offers scripting capabilities, thus
enabling to simulate asynchronous multithreaded simultaneous users to the application. This is the
first stepping stone for us.
Once the load is simulated closed to real behavior, various other tools will be used to identify
application behavior and other environmental factors undergoing changes (as described in
Section 3)
Below table enlists tools which will be used for Performance Testing as well as other activities (e.g.
resources monitoring) during performance testing phase:
Tool Purpose
Visual Studio 2013 For Performance Testing of areas which can’t be worked out
with HP PC / LR.
HP LoadRunner 12 Automated performance and stress testing
NeoLoad Trace client/server communication
SolarWind Defects logging and tracking
Spotlight Enterprise Resource Monitoring
Windows Profiler Extensive Resource Monitoring
Table 2. Selected tools for Performance Testing phases
4.2 SWOT Analysis
In concluding this research, we’ve learned about numerous testing tools, what those are capable
of and their approach towards load simulation.
HP LoadRunner is a market leader in performance testing tools, with over 90% market share
globally. ChromeIS has mature strong expertise in HP LoadRunner and HP Performance Center and
that had always been strength of ChromeIS to address any performance concerns in timely
fashion.
Unfortunately, HP LoadRunner has very limited support for .net protocol and it induces a lot of errors
whilst preparing performance test scripts for OUR PRODUCT. Experiments show that the tool fails in
deserialization, decoding and parsing already desterilized fragments of scripts. We can conclude
that LoadRunner 12 does not offer what it takes to test a sophisticated .net based application with
complex architecture and modern clients.
Time has changed and ChromeIS has taken initiatives to re-invent the products using state of the
art technologies making Rich Internet Applications and optimizing performance using Smart Client.
While this sounds exciting, it brings unforeseen challenges in many ways. Performance and
efficiency of software is undoubtedly among most critical aspects of software and this cannot be
ascertained without undergoing aggressive performance test. Hence, a workaround has been
suggested and Microsoft Visual Studio 2013 will be used for conducting performance testing.
Copyrights © NetSol Technologies Ltd. All rights reserved 30
Unauthorized copy or use of this document is strictly prohibited
Software Quality Assurance Plan
Performance Testing – Tool Selection
4.3 Training Needs
ChromeIS has mature expertise in HP LoadRunner and HP Performance Center; however, these
tools are out of option.
Since ChromeIS will be using Visual Studio 2013 for development of Performance test agents,
certain level of resource training is required. A team size of 4-6 dedicated resources is expected to
be working for Performance Testing. This team will be briefly trained by QAM on performance
testing goals, objectives and overall approach towards meeting quantitative goals. While the
conceptual and quantitative goals remain the same, working with the tool itself will be crucial to
achieve test objectives.
Training programs to be provided to the team to enable successful planning and execution. A
training plan can be published separately or as an addendum to this document.
4.4 Lessons Learnt
Selection of tool and technology for carrying out performance testing is of utmost significance. The
same applies to overall software development as well.
We’ve seen in past as well that development technology was picked without consensus, or even
consultation with Performance Testing team. Now we’ve understood that performance testing an
application involves sophisticated activities, induces unforeseen technical challenges, and
imposes technological constraints, thus essentially making it a challenge to workout performance
testing. This cannot be simply compared with business testing where there is no or little involvement
of technological constrains.
It is suggested to keep performance testing team in due consultation when making technology
selection or making technical changes to existing application (upgrade of Silverlight version,
bringing parts of application logic to thick client). Although the final decision of picking the
development technology or modifying underlying components will be under sole discretion of the
PM, this discussion will ensure any possible constraints are communicated and any possible risks are
addressed
5. Definition of Terms, Acronyms And Abbreviations
Term Description
DEV Development Team
PT Performance Testing
SQA Software Quality Assurance
Copyrights © NetSol Technologies Ltd. All rights reserved 31
Unauthorized copy or use of this document is strictly prohibited

More Related Content

What's hot

Domain Driven Design
Domain Driven Design Domain Driven Design
Domain Driven Design
Araf Karsh Hamid
 
Microservices in the Apache Kafka Ecosystem
Microservices in the Apache Kafka EcosystemMicroservices in the Apache Kafka Ecosystem
Microservices in the Apache Kafka Ecosystem
confluent
 
Battle of the Stream Processing Titans – Flink versus RisingWave
Battle of the Stream Processing Titans – Flink versus RisingWaveBattle of the Stream Processing Titans – Flink versus RisingWave
Battle of the Stream Processing Titans – Flink versus RisingWave
Yingjun Wu
 
Big Data Redis Mongodb Dynamodb Sharding
Big Data Redis Mongodb Dynamodb ShardingBig Data Redis Mongodb Dynamodb Sharding
Big Data Redis Mongodb Dynamodb Sharding
Araf Karsh Hamid
 
Liferay as a Microservice Platform
Liferay as a Microservice PlatformLiferay as a Microservice Platform
Liferay as a Microservice Platform
Daniel Reuther
 
Microservices, Containers, Kubernetes, Kafka, Kanban
Microservices, Containers, Kubernetes, Kafka, KanbanMicroservices, Containers, Kubernetes, Kafka, Kanban
Microservices, Containers, Kubernetes, Kafka, Kanban
Araf Karsh Hamid
 
Webinar "Communication Between Loosely Coupled Microservices"
Webinar "Communication Between Loosely Coupled Microservices"Webinar "Communication Between Loosely Coupled Microservices"
Webinar "Communication Between Loosely Coupled Microservices"
Bernd Ruecker
 
Agile, User Stories, Domain Driven Design
Agile, User Stories, Domain Driven DesignAgile, User Stories, Domain Driven Design
Agile, User Stories, Domain Driven Design
Araf Karsh Hamid
 
Mastering Chaos - A Netflix Guide to Microservices
Mastering Chaos - A Netflix Guide to MicroservicesMastering Chaos - A Netflix Guide to Microservices
Mastering Chaos - A Netflix Guide to Microservices
Josh Evans
 
Microservice architecture design principles
Microservice architecture design principlesMicroservice architecture design principles
Microservice architecture design principles
Sanjoy Kumar Roy
 
Mainframe Integration, Offloading and Replacement with Apache Kafka
Mainframe Integration, Offloading and Replacement with Apache KafkaMainframe Integration, Offloading and Replacement with Apache Kafka
Mainframe Integration, Offloading and Replacement with Apache Kafka
Kai Wähner
 
Hexagonal architecture with Spring Boot
Hexagonal architecture with Spring BootHexagonal architecture with Spring Boot
Hexagonal architecture with Spring Boot
Mikalai Alimenkou
 
Scrum + bdd + ddd
Scrum + bdd + dddScrum + bdd + ddd
Scrum + bdd + ddd
Helder De Oliveira
 
Complex event flows in distributed systems
Complex event flows in distributed systemsComplex event flows in distributed systems
Complex event flows in distributed systems
Bernd Ruecker
 
Apache Flink @ NYC Flink Meetup
Apache Flink @ NYC Flink MeetupApache Flink @ NYC Flink Meetup
Apache Flink @ NYC Flink Meetup
Stephan Ewen
 
Kafka Streams: What it is, and how to use it?
Kafka Streams: What it is, and how to use it?Kafka Streams: What it is, and how to use it?
Kafka Streams: What it is, and how to use it?
confluent
 
A Self-Service API Portal for Developers
A Self-Service API Portal for DevelopersA Self-Service API Portal for Developers
A Self-Service API Portal for Developers
CA Technologies
 
Qlik and Confluent Success Stories with Kafka - How Generali and Skechers Kee...
Qlik and Confluent Success Stories with Kafka - How Generali and Skechers Kee...Qlik and Confluent Success Stories with Kafka - How Generali and Skechers Kee...
Qlik and Confluent Success Stories with Kafka - How Generali and Skechers Kee...
HostedbyConfluent
 
Kafka 101 and Developer Best Practices
Kafka 101 and Developer Best PracticesKafka 101 and Developer Best Practices
Kafka 101 and Developer Best Practices
confluent
 
Mainframe Modernization with Precisely and Microsoft Azure
Mainframe Modernization with Precisely and Microsoft AzureMainframe Modernization with Precisely and Microsoft Azure
Mainframe Modernization with Precisely and Microsoft Azure
Precisely
 

What's hot (20)

Domain Driven Design
Domain Driven Design Domain Driven Design
Domain Driven Design
 
Microservices in the Apache Kafka Ecosystem
Microservices in the Apache Kafka EcosystemMicroservices in the Apache Kafka Ecosystem
Microservices in the Apache Kafka Ecosystem
 
Battle of the Stream Processing Titans – Flink versus RisingWave
Battle of the Stream Processing Titans – Flink versus RisingWaveBattle of the Stream Processing Titans – Flink versus RisingWave
Battle of the Stream Processing Titans – Flink versus RisingWave
 
Big Data Redis Mongodb Dynamodb Sharding
Big Data Redis Mongodb Dynamodb ShardingBig Data Redis Mongodb Dynamodb Sharding
Big Data Redis Mongodb Dynamodb Sharding
 
Liferay as a Microservice Platform
Liferay as a Microservice PlatformLiferay as a Microservice Platform
Liferay as a Microservice Platform
 
Microservices, Containers, Kubernetes, Kafka, Kanban
Microservices, Containers, Kubernetes, Kafka, KanbanMicroservices, Containers, Kubernetes, Kafka, Kanban
Microservices, Containers, Kubernetes, Kafka, Kanban
 
Webinar "Communication Between Loosely Coupled Microservices"
Webinar "Communication Between Loosely Coupled Microservices"Webinar "Communication Between Loosely Coupled Microservices"
Webinar "Communication Between Loosely Coupled Microservices"
 
Agile, User Stories, Domain Driven Design
Agile, User Stories, Domain Driven DesignAgile, User Stories, Domain Driven Design
Agile, User Stories, Domain Driven Design
 
Mastering Chaos - A Netflix Guide to Microservices
Mastering Chaos - A Netflix Guide to MicroservicesMastering Chaos - A Netflix Guide to Microservices
Mastering Chaos - A Netflix Guide to Microservices
 
Microservice architecture design principles
Microservice architecture design principlesMicroservice architecture design principles
Microservice architecture design principles
 
Mainframe Integration, Offloading and Replacement with Apache Kafka
Mainframe Integration, Offloading and Replacement with Apache KafkaMainframe Integration, Offloading and Replacement with Apache Kafka
Mainframe Integration, Offloading and Replacement with Apache Kafka
 
Hexagonal architecture with Spring Boot
Hexagonal architecture with Spring BootHexagonal architecture with Spring Boot
Hexagonal architecture with Spring Boot
 
Scrum + bdd + ddd
Scrum + bdd + dddScrum + bdd + ddd
Scrum + bdd + ddd
 
Complex event flows in distributed systems
Complex event flows in distributed systemsComplex event flows in distributed systems
Complex event flows in distributed systems
 
Apache Flink @ NYC Flink Meetup
Apache Flink @ NYC Flink MeetupApache Flink @ NYC Flink Meetup
Apache Flink @ NYC Flink Meetup
 
Kafka Streams: What it is, and how to use it?
Kafka Streams: What it is, and how to use it?Kafka Streams: What it is, and how to use it?
Kafka Streams: What it is, and how to use it?
 
A Self-Service API Portal for Developers
A Self-Service API Portal for DevelopersA Self-Service API Portal for Developers
A Self-Service API Portal for Developers
 
Qlik and Confluent Success Stories with Kafka - How Generali and Skechers Kee...
Qlik and Confluent Success Stories with Kafka - How Generali and Skechers Kee...Qlik and Confluent Success Stories with Kafka - How Generali and Skechers Kee...
Qlik and Confluent Success Stories with Kafka - How Generali and Skechers Kee...
 
Kafka 101 and Developer Best Practices
Kafka 101 and Developer Best PracticesKafka 101 and Developer Best Practices
Kafka 101 and Developer Best Practices
 
Mainframe Modernization with Precisely and Microsoft Azure
Mainframe Modernization with Precisely and Microsoft AzureMainframe Modernization with Precisely and Microsoft Azure
Mainframe Modernization with Precisely and Microsoft Azure
 

Viewers also liked

Loadrunner vs Jmeter
Loadrunner vs JmeterLoadrunner vs Jmeter
Loadrunner vs Jmeter
Atul Pant
 
Testing Soap/Rest Web Services with JMeter
Testing Soap/Rest Web Services with JMeterTesting Soap/Rest Web Services with JMeter
Testing Soap/Rest Web Services with JMeter
BlazeMeter
 
Gender in on-line communities: StackOverflow, WordPress, Drupal
Gender in on-line communities: StackOverflow, WordPress, DrupalGender in on-line communities: StackOverflow, WordPress, Drupal
Gender in on-line communities: StackOverflow, WordPress, Drupal
Alexander Serebrenik
 
ไตร่ตรองงานวิจัยของฉัน
ไตร่ตรองงานวิจัยของฉันไตร่ตรองงานวิจัยของฉัน
ไตร่ตรองงานวิจัยของฉัน
School in Phatthalung
 
Roman Vorobyev
Roman VorobyevRoman Vorobyev
Roman Vorobyev
Roman Vorobyev
 
Spain
SpainSpain
Car Study &amp; Statistics
Car Study &amp; StatisticsCar Study &amp; Statistics
Car Study &amp; Statistics
Melissa Anne Lim
 
researchpbl
researchpblresearchpbl
researchpbl
puniga
 
TTT
TTTTTT
TTT
Razvain
 
TTT
TTTTTT
TTT
Razvain
 
Benevol 2013: Visualizing the complexity of software module upgrades
Benevol 2013: Visualizing the complexity of software module upgradesBenevol 2013: Visualizing the complexity of software module upgrades
Benevol 2013: Visualizing the complexity of software module upgrades
Alexander Serebrenik
 
865 social capital
865 social capital865 social capital
865 social capital
Ian Pollock
 
Fresh Produce
Fresh ProduceFresh Produce
Fresh Produce
Colt
 
An empirical study of the evolution of Eclipse third-party plug-ins
An empirical study of the evolution of Eclipse third-party plug-insAn empirical study of the evolution of Eclipse third-party plug-ins
An empirical study of the evolution of Eclipse third-party plug-ins
Alexander Serebrenik
 
Hh kehittamistyo esitys_atte_jarvela
Hh kehittamistyo esitys_atte_jarvelaHh kehittamistyo esitys_atte_jarvela
Hh kehittamistyo esitys_atte_jarvelaAtte Järvelä
 
Future trends in technology
Future trends in technologyFuture trends in technology
Future trends in technology
Dardanelle Public Schools
 
Invited Talk MESOCA 2014: Evolving software systems: emerging trends and chal...
Invited Talk MESOCA 2014: Evolving software systems: emerging trends and chal...Invited Talk MESOCA 2014: Evolving software systems: emerging trends and chal...
Invited Talk MESOCA 2014: Evolving software systems: emerging trends and chal...
Alexander Serebrenik
 
Metropolia - Projektityön esitys
Metropolia - Projektityön esitysMetropolia - Projektityön esitys
Metropolia - Projektityön esitys
Atte Järvelä
 
Challenges in Software Ecosystems Research
Challenges in Software Ecosystems ResearchChallenges in Software Ecosystems Research
Challenges in Software Ecosystems Research
Alexander Serebrenik
 
Reunió pares p3 juny curs 12 13
Reunió pares p3 juny  curs 12 13Reunió pares p3 juny  curs 12 13
Reunió pares p3 juny curs 12 13edumallol
 

Viewers also liked (20)

Loadrunner vs Jmeter
Loadrunner vs JmeterLoadrunner vs Jmeter
Loadrunner vs Jmeter
 
Testing Soap/Rest Web Services with JMeter
Testing Soap/Rest Web Services with JMeterTesting Soap/Rest Web Services with JMeter
Testing Soap/Rest Web Services with JMeter
 
Gender in on-line communities: StackOverflow, WordPress, Drupal
Gender in on-line communities: StackOverflow, WordPress, DrupalGender in on-line communities: StackOverflow, WordPress, Drupal
Gender in on-line communities: StackOverflow, WordPress, Drupal
 
ไตร่ตรองงานวิจัยของฉัน
ไตร่ตรองงานวิจัยของฉันไตร่ตรองงานวิจัยของฉัน
ไตร่ตรองงานวิจัยของฉัน
 
Roman Vorobyev
Roman VorobyevRoman Vorobyev
Roman Vorobyev
 
Spain
SpainSpain
Spain
 
Car Study &amp; Statistics
Car Study &amp; StatisticsCar Study &amp; Statistics
Car Study &amp; Statistics
 
researchpbl
researchpblresearchpbl
researchpbl
 
TTT
TTTTTT
TTT
 
TTT
TTTTTT
TTT
 
Benevol 2013: Visualizing the complexity of software module upgrades
Benevol 2013: Visualizing the complexity of software module upgradesBenevol 2013: Visualizing the complexity of software module upgrades
Benevol 2013: Visualizing the complexity of software module upgrades
 
865 social capital
865 social capital865 social capital
865 social capital
 
Fresh Produce
Fresh ProduceFresh Produce
Fresh Produce
 
An empirical study of the evolution of Eclipse third-party plug-ins
An empirical study of the evolution of Eclipse third-party plug-insAn empirical study of the evolution of Eclipse third-party plug-ins
An empirical study of the evolution of Eclipse third-party plug-ins
 
Hh kehittamistyo esitys_atte_jarvela
Hh kehittamistyo esitys_atte_jarvelaHh kehittamistyo esitys_atte_jarvela
Hh kehittamistyo esitys_atte_jarvela
 
Future trends in technology
Future trends in technologyFuture trends in technology
Future trends in technology
 
Invited Talk MESOCA 2014: Evolving software systems: emerging trends and chal...
Invited Talk MESOCA 2014: Evolving software systems: emerging trends and chal...Invited Talk MESOCA 2014: Evolving software systems: emerging trends and chal...
Invited Talk MESOCA 2014: Evolving software systems: emerging trends and chal...
 
Metropolia - Projektityön esitys
Metropolia - Projektityön esitysMetropolia - Projektityön esitys
Metropolia - Projektityön esitys
 
Challenges in Software Ecosystems Research
Challenges in Software Ecosystems ResearchChallenges in Software Ecosystems Research
Challenges in Software Ecosystems Research
 
Reunió pares p3 juny curs 12 13
Reunió pares p3 juny  curs 12 13Reunió pares p3 juny  curs 12 13
Reunió pares p3 juny curs 12 13
 

Similar to Performance Test WCF/WPF app - Selecting right Tool

Guidelines For The Issuance And Management Of EV Code Signing Certificate
Guidelines For The Issuance And Management Of EV Code Signing CertificateGuidelines For The Issuance And Management Of EV Code Signing Certificate
Guidelines For The Issuance And Management Of EV Code Signing Certificate
CodeSigningStore
 
Peoplesoft AWE_hcm_delegation_framework
Peoplesoft  AWE_hcm_delegation_frameworkPeoplesoft  AWE_hcm_delegation_framework
Peoplesoft AWE_hcm_delegation_framework
Purna Chandra
 
B12303
B12303B12303
B12303
rajuuuu
 
Install
InstallInstall
Install
Said Chatir
 
iFobs Manual
iFobs ManualiFobs Manual
iFobs Manual
Den Usenko
 
Rman
RmanRman
1 Pdfsam
1 Pdfsam1 Pdfsam
1 Pdfsam
Emanuel Mateus
 
1 Rac
1 Rac1 Rac
Developingsecurewebappswatchfire
DevelopingsecurewebappswatchfireDevelopingsecurewebappswatchfire
Developingsecurewebappswatchfire
Aung Khant
 
B13922
B13922B13922
Installing and conf guide for hp sm connector
Installing and conf guide for hp sm connectorInstalling and conf guide for hp sm connector
Installing and conf guide for hp sm connector
TheEnferRimbaud
 
Transmission resource management
Transmission resource managementTransmission resource management
Transmission resource management
yascisse
 
Admin
AdminAdmin
Adf tutorial oracle
Adf tutorial oracleAdf tutorial oracle
Adf tutorial oracle
César Augusto Castillo Farfán
 
Oracle performance tuning
Oracle performance tuningOracle performance tuning
Oracle performance tuning
vksgarg
 
Oracle forms and resports
Oracle forms and resportsOracle forms and resports
Oracle forms and resports
pawansharma1986
 
Subversion howto
Subversion howtoSubversion howto
0859 - Update White Paper Choosing Linux for Medical Devices_NO_cover
0859 - Update White Paper Choosing Linux for Medical Devices_NO_cover 0859 - Update White Paper Choosing Linux for Medical Devices_NO_cover
0859 - Update White Paper Choosing Linux for Medical Devices_NO_cover
Ken Herold
 
Party merge
Party mergeParty merge
Party merge
shameem_fics
 
Forwarding Connector Configuration Guide 7.0.7.7286
Forwarding Connector Configuration Guide 7.0.7.7286Forwarding Connector Configuration Guide 7.0.7.7286
Forwarding Connector Configuration Guide 7.0.7.7286
Protect724gopi
 

Similar to Performance Test WCF/WPF app - Selecting right Tool (20)

Guidelines For The Issuance And Management Of EV Code Signing Certificate
Guidelines For The Issuance And Management Of EV Code Signing CertificateGuidelines For The Issuance And Management Of EV Code Signing Certificate
Guidelines For The Issuance And Management Of EV Code Signing Certificate
 
Peoplesoft AWE_hcm_delegation_framework
Peoplesoft  AWE_hcm_delegation_frameworkPeoplesoft  AWE_hcm_delegation_framework
Peoplesoft AWE_hcm_delegation_framework
 
B12303
B12303B12303
B12303
 
Install
InstallInstall
Install
 
iFobs Manual
iFobs ManualiFobs Manual
iFobs Manual
 
Rman
RmanRman
Rman
 
1 Pdfsam
1 Pdfsam1 Pdfsam
1 Pdfsam
 
1 Rac
1 Rac1 Rac
1 Rac
 
Developingsecurewebappswatchfire
DevelopingsecurewebappswatchfireDevelopingsecurewebappswatchfire
Developingsecurewebappswatchfire
 
B13922
B13922B13922
B13922
 
Installing and conf guide for hp sm connector
Installing and conf guide for hp sm connectorInstalling and conf guide for hp sm connector
Installing and conf guide for hp sm connector
 
Transmission resource management
Transmission resource managementTransmission resource management
Transmission resource management
 
Admin
AdminAdmin
Admin
 
Adf tutorial oracle
Adf tutorial oracleAdf tutorial oracle
Adf tutorial oracle
 
Oracle performance tuning
Oracle performance tuningOracle performance tuning
Oracle performance tuning
 
Oracle forms and resports
Oracle forms and resportsOracle forms and resports
Oracle forms and resports
 
Subversion howto
Subversion howtoSubversion howto
Subversion howto
 
0859 - Update White Paper Choosing Linux for Medical Devices_NO_cover
0859 - Update White Paper Choosing Linux for Medical Devices_NO_cover 0859 - Update White Paper Choosing Linux for Medical Devices_NO_cover
0859 - Update White Paper Choosing Linux for Medical Devices_NO_cover
 
Party merge
Party mergeParty merge
Party merge
 
Forwarding Connector Configuration Guide 7.0.7.7286
Forwarding Connector Configuration Guide 7.0.7.7286Forwarding Connector Configuration Guide 7.0.7.7286
Forwarding Connector Configuration Guide 7.0.7.7286
 

Recently uploaded

“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
Edge AI and Vision Alliance
 
Skybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoptionSkybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoption
Tatiana Kojar
 
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance PanelsNorthern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving
 
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptxPRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
christinelarrosa
 
Session 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdfSession 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdf
UiPathCommunity
 
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
saastr
 
A Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's ArchitectureA Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's Architecture
ScyllaDB
 
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin..."$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
Fwdays
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
Antonios Katsarakis
 
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptxThe Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
operationspcvita
 
Christine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptxChristine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptx
christinelarrosa
 
Apps Break Data
Apps Break DataApps Break Data
Apps Break Data
Ivo Velitchkov
 
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Pitangent Analytics & Technology Solutions Pvt. Ltd
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
Fwdays
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
MichaelKnudsen27
 
Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
c5vrf27qcz
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
Fwdays
 
Mutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented ChatbotsMutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented Chatbots
Pablo Gómez Abajo
 
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-EfficiencyFreshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
ScyllaDB
 
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users",  Kevin Goedecke"Scaling RAG Applications to serve millions of users",  Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
Fwdays
 

Recently uploaded (20)

“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
 
Skybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoptionSkybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoption
 
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance PanelsNorthern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
 
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptxPRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
 
Session 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdfSession 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdf
 
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
 
A Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's ArchitectureA Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's Architecture
 
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin..."$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
 
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptxThe Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
 
Christine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptxChristine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptx
 
Apps Break Data
Apps Break DataApps Break Data
Apps Break Data
 
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
 
Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
 
Mutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented ChatbotsMutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented Chatbots
 
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-EfficiencyFreshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
 
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users",  Kevin Goedecke"Scaling RAG Applications to serve millions of users",  Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
 

Performance Test WCF/WPF app - Selecting right Tool

  • 1. Software Quality Assurance Plan For WCF/WPF based Application Version 0.1 May 15, 2014 Compiled by: Kamran Shaukat Ali Khan Chrome Technologies ChromeIS.com
  • 2. Software Quality Assurance Plan Performance Test Plan – Ascent –Version 0.1 Table of Contents Document Information................................................................................................vi Document Information.................................................................................................vi Document Review Information.....................................................................................vi Document Review Information.....................................................................................vi Document Revision History..........................................................................................vi Document Revision History...........................................................................................vi List of Figures..........................................................................................................vii List of Figures.............................................................................................................vii List of Tables............................................................................................................vii List of Tables...............................................................................................................vii 1. Abstract.....................................................................................................................1 1.1 Proposed Solution...................................................................................................1 2. Performance Testing Tools Evaluated.........................................................................2 2.1 HP LoadRunner and Performance Center....................................................................2 2.2 IBM Rational Performance Tester..............................................................................3 2.3 Borland Silk Performer............................................................................................3 2.4 Telerik Test Studio..................................................................................................3 2.5 Apache JMeter.......................................................................................................3 2.6 j-Chav..................................................................................................................4 2.7 Microsoft WCAT – Web Capacity Analysis Tool (formerly WAS – Web Application Stress Tool)..........................................................................................................................4 2.8 Sun VisualVM.........................................................................................................4 2.9 WAPT...................................................................................................................5 2.10 Web Application Load Simulator (LoadSim)...............................................................5 2.11 Neo Load.............................................................................................................5 2.12 EggPlant Load Tester.............................................................................................5 2.13 OpenSTA – Open System Testing Architecture .........................................................6 2.14 OpenWebLoad......................................................................................................6 2.15 OpenLoad............................................................................................................6 2.16 loadUI.................................................................................................................6 2.17 LoadeaTest..........................................................................................................6 2.18 LoadManager.......................................................................................................7 2.19 Test Automation FX...............................................................................................7 2.20 CodedUI..............................................................................................................7 2.21 Soap UI...............................................................................................................7 2.22 Soap-Stone.........................................................................................................7 2.23 Web Performance Suite.........................................................................................8 2.24 Webserver Stress Tool...........................................................................................8 2.25 WebLOAD............................................................................................................8 2.26 QALoad...............................................................................................................8 2.27 Lisa Load.............................................................................................................8 2.28 Grinder...............................................................................................................9 2.29 nGrinder..............................................................................................................9 Copyrights © NetSol Technologies Ltd. All rights reserved iii Unauthorized copy or use of this document is strictly prohibited
  • 3. Software Quality Assurance Plan Performance Test Plan – Ascent –Version 0.1 2.30 GrinderStone.......................................................................................................9 2.31 MStone...............................................................................................................9 2.32 ValGrind..............................................................................................................9 2.33 QEngine..............................................................................................................9 2.34 Qtest................................................................................................................10 2.35 HammerHead 2..................................................................................................10 2.36 Hammerora........................................................................................................10 2.37 Pylot.................................................................................................................10 2.38 OpenWebLoad....................................................................................................10 2.39 SD’s Profiler.......................................................................................................11 2.40 Benerator..........................................................................................................11 2.41 SOATest (by Parasoft).........................................................................................11 2.42 Scapa Test and Performance Platform ...................................................................11 2.43 CLIF Load Injection ............................................................................................12 2.44 ContiPerf...........................................................................................................12 2.45 Coveriety...........................................................................................................12 2.46 D-ITG (Distributed Internet Traffic Generator).........................................................12 2.47 Database Opensource Test Suit (DOTS).................................................................13 2.48 DB Monster........................................................................................................13 2.49 Deluge..............................................................................................................13 2.50 DieselTest..........................................................................................................13 2.51 Faban................................................................................................................13 2.52 FunkLoad...........................................................................................................14 2.53 Curl-Loader........................................................................................................14 2.54 Spirant Avalanche...............................................................................................14 2.55 IXIA IxLoad........................................................................................................14 2.56 Paessler.............................................................................................................14 2.57 AQ TestComplete and AQ Time.............................................................................14 2.58 Test Maker.........................................................................................................15 2.59 TP Test..............................................................................................................15 2.60 Tsung................................................................................................................15 2.61 Tnsgen..............................................................................................................15 2.62 N-Time..............................................................................................................15 2.63 p-unit................................................................................................................16 2.64 Lobo.................................................................................................................16 2.65 MessAdmin........................................................................................................16 2.66 Multi Router Traffic Grapher (MRTR)......................................................................16 2.67 RRRD Tool.........................................................................................................16 2.68 Multi-Mechanize..................................................................................................16 2.69 Stress Driver......................................................................................................17 2.70 Load Impact.......................................................................................................17 2.71 Pure Load..........................................................................................................17 2.72 Pure Agent.........................................................................................................17 2.73 FWPTT...............................................................................................................17 2.74 JBlits Pro...........................................................................................................18 2.75 JCrawler............................................................................................................18 Copyrights © NetSol Technologies Ltd. All rights reserved iv Unauthorized copy or use of this document is strictly prohibited
  • 4. Software Quality Assurance Plan Performance Test Plan – Ascent –Version 0.1 2.76 PandoraFMS.......................................................................................................18 2.77 Siege................................................................................................................18 2.78 HTTP Load.........................................................................................................18 2.79 httPerf...............................................................................................................19 2.80 iPerf..................................................................................................................19 2.81 Site Blaster........................................................................................................19 2.82 RawLoadTester...................................................................................................19 2.83 Allmon..............................................................................................................20 2.84 AppsWatch.........................................................................................................20 2.85 BurnIn Test........................................................................................................20 2.86 Capacity Calibration (CapCal)...............................................................................20 2.87 Chariot..............................................................................................................21 2.88 CitraTestVU........................................................................................................21 2.89 SSW Performance Pro..........................................................................................21 2.90 e-Load..............................................................................................................21 2.91 MLoad...............................................................................................................21 2.92 Monitor Master...................................................................................................22 2.93 Forecast............................................................................................................22 2.94 GS DataGenerator...............................................................................................22 2.95 vPerformer.........................................................................................................22 2.96 JUnit.................................................................................................................22 2.97 Xceptance LoadTest............................................................................................23 2.98 zapwireless........................................................................................................23 2.99 Seagull..............................................................................................................23 2.100 SIPp................................................................................................................23 2.101 SLAMD............................................................................................................23 2.102 IxoraRMS.........................................................................................................24 2.103 j-hawk.............................................................................................................24 2.104 Wireshark........................................................................................................24 2.105 Ostinato...........................................................................................................24 2.106 Eclipse Test & Performance Tools Platform Project (EPTP)........................................24 2.107 Web Polygraph.................................................................................................24 3. Tools Selection Criteria.............................................................................................25 3.1 Business and Technical Requirements......................................................................25 3.2 Performance Test Objectives..................................................................................25 3.3 Performance Test Execution...................................................................................28 3.4 Load Description...................................................................................................28 4. Conclusion................................................................................................................29 4.1 Selected Tools......................................................................................................30 4.2 SWOT Analysis.....................................................................................................30 4.3 Training Needs.....................................................................................................31 4.4 Lessons Learnt.....................................................................................................31 5. Definition of Terms, Acronyms And Abbreviations.....................................................31 Copyrights © NetSol Technologies Ltd. All rights reserved v Unauthorized copy or use of this document is strictly prohibited
  • 5. Software Quality Assurance Plan Performance Test Plan – Ascent –Version 0.1 Document & Data Control Document InformationDocument Information Category Information <Title> Performance Test Tool Selection <Version> 0.1 <Author> Kamran Shaukat Ali Khan <Identifier> <Life Cycle Activity> TLC <Issue Date> Pending Document Review InformationDocument Review Information Review Date Reviewer Name Version Reference / Evidence Document Revision HistoryDocument Revision History Author Date Version Description Kamran Shaukat Ali Khan 15 May 2014 0.1 Draft Copyrights © NetSol Technologies Ltd. All rights reserved vi Unauthorized copy or use of this document is strictly prohibited
  • 6. Software Quality Assurance Plan Performance Test Plan – Ascent –Version 0.1 List of FiguresList of Figures Figure 1, Performance Acceptance at a glance..............................................................27 List of TablesList of Tables Table 1. Tools fostering Performance Testing phases.....................................................2 Table 2. Selected tools for Performance Testing phases................................................30 Copyrights © NetSol Technologies Ltd. All rights reserved vii Unauthorized copy or use of this document is strictly prohibited
  • 7. Software Quality Assurance Plan Performance Testing – Tool Selection 1. Abstract OUR PRODUCT is engineered for performance and agility. Performance testing is the process of assessing performance/efficiency of a transaction system whilst simultaneous users are accessing it; this is to determine if subject application (with all factors involved) can serve its purpose efficiently. Performance Testing is a fundamental aspect of software engineering, but it is a practice that too often is easily forgotten in today’s fast-paced application development culture. Performance testing applications efficiently is essentially an integral component for meeting end customer expectations. The aim of this document is to evaluate and compare all available performance testing tools to determine their usability and effectiveness. Software testing is a crucial part of software development and the process of automating software testing is vital to its success. OUR PRODUCT is a complex system and performance testing throughout the software development cycle is pertinent to the success of the software. To enable comparison of the automated performance testing tools, a multi-partitioned metric suite was developed to guide the tool evaluation process. Inferences, implications and results are presented and discussed in Section 3. 1.1 Proposed Solution The overall goal of this research is to evaluate and compare several automated performance testing tools to determine their usability and effectiveness for OUR PRODUCT. To accomplish this goal, the research: 1. Prepared a comprehensive performance testing tools available on internet. 2. Broadened the scope of tools to include shareware, freeware, Open licenses and open source tools. The list of tools, along with tools’ assessment is described in Section 2. 3. Defined criteria for tools selection based on business and technical requirements of OUR PRODUCT; this will help narrow down the scope of research. 4. Compared existing tools in practice at ChromeIS with evolving requirements as highlighted in above point. 5. Performed assessment of tools based on features and later, experiment with actual application i.e. OUR PRODUCT (for e.g. CMS) 6. Tested the target applications using the selected automated testing tools and gather resulting data 7. Evaluated and interpret results to draw inferences and make recommendations OUR PRODUCT is a smart client being developed in Visual Studio 2010 using.net framework 4.5 – this imposes certain limitations. An RND activity has been concluded where a number of tools were evaluated. The assessment concluded that Performance Testing has to be carried out in Visual Studio .net itself. However, there are certain other tools which can be utilized to foster in Performance Testing. Below table enlists tools which will be used for Performance Testing as well as for resources monitoring: Copyrights © NetSol Technologies Ltd. All rights reserved 1 Unauthorized copy or use of this document is strictly prohibited
  • 8. Software Quality Assurance Plan Performance Testing – Tool Selection Tool Purpose Visual Studio 2013 For Performance Testing of areas which can’t be worked out with HP PC / LR. HP LoadRunner 12 Automated performance and stress testing NeoLoad Trace client/server communication SolarWind Defects logging and tracking Spotlight Enterprise Resource Monitoring Windows Profiler Extensive Resource Monitoring Table 1. Tools fostering Performance Testing phases 2. Performance Testing Tools Evaluated The tools summarized in this section constitute a concise list of tools available over internet for automated performing testing (over a hundred tools were looked into). Each tool was evaluated based on the criteria defined in Section 3. 2.1 HP LoadRunner and Performance Center HP LoadRunner is a cost-effective and comprehensive tool used for automated performance testing. This is our first choice because: • LR is industry leader with over 90% market share • ChromeIS has most experience in HP LR/PC • Fully trained resources available • Uses standard programming techniques and common languages such as C, VB and Java; thus eliminating need to learn new language. • Easy to script, parameterize, correlate and close to real test execution • Extensive reporting, graphs, statistics collection, helping in root cause identification HP LoadRunner brought support for .net protocol in version 12 which was released last month (15 th April 2014); however, the support (as in readme file) does not describe component level details which can be recognized by LoadRunner. Through experiments, we can conclude that LR does not recognize OUR PRODUCT CMS client. It fails with the most basic level of scripting (login only – without selecting a branch) and the LoadRunner component i.e. VUGen stops responding. The script generated fails to compile and introduces number of errors. We can keep an eye on new patches of LR and keep experiments alive. However, this must not stop overall performance test progress. As of now, the tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. Copyrights © NetSol Technologies Ltd. All rights reserved 2 Unauthorized copy or use of this document is strictly prohibited
  • 9. Software Quality Assurance Plan Performance Testing – Tool Selection 2.2 IBM Rational Performance Tester IBM RPT was 2nd choice since they hold nearly 6% out of 10% market share (after 90% being with LR). IBM RPT is slow in scripting, intelligence, execution and other aspects. However, nowhere on their website IBM claims that RPT supports .net or WCF or WPF. A formal conversation was opened with IBM but there had been no response. We can keep an eye on IBM newsletter to see if they provide any sophisticated mechanism for performance testing of WCF or SOA based architecture. As of now, the tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.3 Borland Silk Performer Silk Performer 15 now enables to capture all the raw measure data of a load test. It now automatically detects asynchronous communication patterns such as polling and long-polling. It more effectively supports applications with periodic, event-based, or server-triggered content updates. Tool is not as rich and introduced support for Ajax just recently with no support for modern needs of RIA based clients and smart clients. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.4 Telerik Test Studio The tool fails to recognize Telerik “ribbon” components. This was discussed with Telerik support and they confirmed the problem exists and listed as known issues. Since the tool fails to recognize the first component which loads after login, the tool cannot be used. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.5 Apache JMeter Through experiments, we can conclude the JMeter is able to script the calls, however, since OUR PRODUCT uses encryption, JMeter fails to decrypt the client/server communication. This is necessary since the JMeter cannot process an encrypted call. Ideally, we require JMeter to decrypt the server response, process it as required, re-crypt the newly formulated request and then send it to the server. JMeter needs to do it for each response from server to simulate load. Another workaround could be to remove encryption from OUR PRODUCT as long as application is undergoing performance testing. Although this imposes possible concern like: • The encryption and decryption is costly activity and requires additional layer of effort • The performance measures may not be realistic (slightly improved than real) since client and server don’t have to encrypt or decrypt Copyrights © NetSol Technologies Ltd. All rights reserved 3 Unauthorized copy or use of this document is strictly prohibited
  • 10. Software Quality Assurance Plan Performance Testing – Tool Selection If the above constraint can be lived with, we can consider JMeter further. 2.6 j-Chav JChav is a way to see the change in performance of a web application over time, by running a benchmark test for each build. JChav reads all the JMeter logs from each run (one per build), and produces a set of charts for each test in each run. The tool compliments JMeter and hence cannot be right fit for OUR PRODUCT. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.7 Microsoft WCAT – Web Capacity Analysis Tool (formerly WAS – Web Application Stress Tool) WCAT (Web Capacity Analysis Tool) is an excellent tiny tool from Microsoft to perform load test web application on IIS. This tool enables users to do performance analysis on various scenarios of web application. All “perfmon” performance counters (like processor time, private bytes usage, disk queue length, total bytes sent or received in network) can be specified for the performance testing. Once scenario in application (i.e. appropriate URLs) is specified, a number of virtual clients from various client machines will visit the URL with appropriate request data. Like other performance tools, the following players are involved in a performance testing: • Server – machines on which your web application components are running • Client – A virtual client on a machine which acts as end-user to visit the URL • Controller – This coordinates a test among the virtual clients on various machines. It also captures and collates the performance counters from appropriate servers. • Works in Visual FoxPro. However, with all the excellence this tool may have, it is unable to recognize OUR PRODUCT application components. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.8 Sun VisualVM VisualVM is a free visual tool, originally from Sun, to monitor and troubleshoot Java applications. Runs on Sun JDK 6, but is able to monitor applications running on JDK 1.4 and higher. Utilizes various available technologies like jvmstat, JMX, the Serviceability Agent (SA), and the Attach API to get data and uses minimal overhead on monitored applications. Capabilities include: automatically detects and lists locally and remotely running Java applications; monitor application performance and memory consumption; profile application performance or analyze memory allocation; is able to save application configuration and runtime environment together with all taken thread dumps, heap dumps and profiler snaphots into a single application snapshot which can be later processed offline. Tool is more focused on java based applications. The tool does not address modern needs of RIA based clients and smart clients. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. Copyrights © NetSol Technologies Ltd. All rights reserved 4 Unauthorized copy or use of this document is strictly prohibited
  • 11. Software Quality Assurance Plan Performance Testing – Tool Selection 2.9 WAPT WAPT can test and analyze the performance characteristics and bottlenecks of web under various load conditions with dynamic content and secure HTTPS pages. Real User Simulation with Dynamic Test Data, Recording and Playback of HTTPS/SSL Pages, Flexible Test Volume & Load Definition, Clear Reports and Graphs, Full Log of Virtual Users Activities Adjustable delays ("think times") between requests to simulate real user behavior Support of redirects, including redirects to another server Support of all types of proxies. The tool is very mature but it lacks support of WPF, WCF and smart clients. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.10 Web Application Load Simulator (LoadSim) LoadSim is a web application load simulator. It allows you to create simulations and have those simulations run against your webserver. It’s a web application load simulator. It allows you to create simulations and have those simulations run against your webserver. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.11 Neo Load NeoLoad is a professional load testing software for web applications (ASP.Net, J2EE, PHP, etc.) Works almost without scripting, automatic handling of ASP.Net VIEWSTATE, Automatic support of user sessions based on cookies or URL rewriting, Realistic VU simulation, cookies handling, plays simultaneously Virtual Users with different behaviors, modem emulation, IP spoofing, SSL support (record and playback), HTML/PDF/XML reporting, Compare results of several tests, Controller and Load Injectors are multi-platform: Windows, Unix. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. However, it successfully transcribes the sequence and composition of calls made to web services. This can be used in case the composition is not known to performance tester. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.12 EggPlant Load Tester eggPlant performance enabled HTTP-oriented Java and .NET Web Virtual Users (VUs) to be easily generated by recording use-cases. eggPlant Performance is designed for asynchronous requests, push technology, HTTP streaming, and dynamic data originated by the client by supporting multiple threads of execution and intelligent script generation. Some applications, however, simply cannot be automated at the HTTP level and are best automated at the GUI application level. In these situations eggPlant Performance can be used with eggPlant Functional or Selenium VUs for application level testing. eggPlant has in-built support for driving common non-web client-side APIs such as .NET and Java. This leverages the native support for security, session management, and data serialization making test creation faster, more effective, and more robust. The tool supports mobile clients, complex thick clients and propriety devices. More information is available. Copyrights © NetSol Technologies Ltd. All rights reserved 5 Unauthorized copy or use of this document is strictly prohibited
  • 12. Software Quality Assurance Plan Performance Testing – Tool Selection The tool needs further RND to reach a conclusion. 2.13 OpenSTA – Open System Testing Architecture OpenSTA is a distributed software testing architecture based on CORBA. Using OpenSTA (Open System Testing Architecture) a user can generate realistic heavy loads simulating the activity of multiple concurrent virtual users. OpenSTA graphs both virtual user response times and resource utilization information from all Web Servers, Application Servers, Database Servers and Operating Platforms within environment, so that precise performance measurements can be gathered during load tests and analysis on these measurements can be performed. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.14 OpenWebLoad OpenWebLoad is a tool for load testing web applications. It aims to be easy to use and providing near real-time performance measurements of the application under test. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.15 OpenLoad OpenLoad provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.16 loadUI loadUI tool focused mainly on Light Directory Access Protocol (LDAP) and web applications working on HTTP, Web Services, REST, AMF, JMS, JDBC as well. The tool appears to be limited in functionalities it offers. The tool is incapable of handling Rich Internet Application (RIA) such as flex or Silverlight. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. 2.17 LoadeaTest LoadeaTest works in C# and .net and has various components. Capture module creates new test, modelize it as a script, enhances, replays and validates it. Control module defines test properties to launch test on several machines, monitors test in real time and gives a results view. An injection module executes script and collect data on different machines. Collect module is an http interface between all modules and database for storage dump. Last, analysis module is to view test results as tables and graphs, to generate reports, and to activate statistic algorithm on data (Principle Component Analysis). It works with HTTP, TCP, SOAP, RSS System resources. Scripts in high level language (C#, VB.NET ) and data in XML format. Copyrights © NetSol Technologies Ltd. All rights reserved 6 Unauthorized copy or use of this document is strictly prohibited
  • 13. Software Quality Assurance Plan Performance Testing – Tool Selection The tool provides no mechanism for WCF specific applications in .net. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.18 LoadManager LoadManager was developed in Java (Eclipse platform). The result is a testing tool which runs on all platforms supported by Eclipse and Java such as Linux, Windows, HP Unix, etc. Supported protocols are: HTTP, HTTPS, Socket (TCP/IP), JMS, RMI, scripts in high level language (C#, VB.NET ) and data in XML format. Color-coding shows which users pass/fail and a script-link is provided* Testing of individual or groups of users can be paused/restarted at will. The tool provides no mechanism for WCF specific applications in .net. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.19 Test Automation FX TBC The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.20 CodedUI TBC The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.21 Soap UI TBC The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.22 Soap-Stone Soap Stone is a Network benchmark application which can put your network under load and conduct automatic benchmark and recording activities The tool is limited to network load. This is very basic to our requirement. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. Copyrights © NetSol Technologies Ltd. All rights reserved 7 Unauthorized copy or use of this document is strictly prohibited
  • 14. Software Quality Assurance Plan Performance Testing – Tool Selection 2.23 Web Performance Suite Web Performance Suite includes a client-side analysis module (Analyzer) a load testing module (Load Tester) and server analysis modules. It is capable of handling most popular web-design technologies such as AJAX, Web forms, J2EE, ASP, .NET/C#, PHP, ColdFusion, Java, etc. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.24 Webserver Stress Tool Webserver Stress Tool is a HTTP-client/server test application The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.25 WebLOAD WebLOAD is an Open Source, available for free download and use, WebLOAD is a commercial- grade open source project with more than 250 engineering years of product development. Companies that require commercial support, additional productivity features and compatibility with third-party protocols have the option of purchasing WebLOAD Professional. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.26 QALoad QALoad is a performance testing tool from Micro Focus used to perform repeatable load testing and determine the ultimate performance and potential limits of a system QALoad has its own language, EasyScript. It has no support for Silverlight or Smart Client. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.27 Lisa Load LISA offers direct functional and performance validation of every component of distributed applications, with or without a UI present. LISA load tests also leverage the end-to-end functional tests created in LISA, invoking and verifying at every tier of the architecture, with a very low system footprint required, so capacity can be simulated at the lowest possible effort and cost. Lisa Load supports only Java/Swing; hence making it as not a choice for OUR PRODUCT. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. Copyrights © NetSol Technologies Ltd. All rights reserved 8 Unauthorized copy or use of this document is strictly prohibited
  • 15. Software Quality Assurance Plan Performance Testing – Tool Selection 2.28 Grinder Grinder tool focused mainly on Light Directory Access Protocol (LDAP) and web applications working on HTTP protocol. The tool appears to be limited in functionalities it offers. The tool is incapable of handling Rich Internet Application (RIA) such as flex or Silverlight. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.29 nGrinder nGrinder is a stress testing tool, executing script creation, test execution, monitoring, and result report generator simultaneously. Since Grinder itself is not applicable to OUR PRODUCT, the GrinderStone is not applicable either. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.30 GrinderStone GrinderStone is an Eclipse plug-in for Grinder load testing scripts development including debugging, modularity and pretty logging. Since Grinder itself is not applicable to OUR PRODUCT, the GrinderStone is not applicable either. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.31 MStone Mstone started as a mail performance measurement system but now can simultaneously test SMTP, POP, IMAP, and some HTML based systems. It measures transaction latency in multiple stages, and graphs the combined results from multiple clients. MStone provider no mechanism for RIA or Smart clients. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.32 ValGrind Valgrind is focused on Linux programs. Built in tools automatically detect many memory management and threading bugs. ValGrind is not robust enough for RIA and smart clients. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.33 QEngine AdventNet QEngine WebTest provides accurate way to test the functionality of Web applications in both Windows and Linux machines. It provides an easy-to-learn user interface to completely automate the creation of sophisticated test scripts, allows record and playback Web browser events in IE and Mozilla browsers. The tool is immature and focused more on Regression automation with no support for Silverlight. Copyrights © NetSol Technologies Ltd. All rights reserved 9 Unauthorized copy or use of this document is strictly prohibited
  • 16. Software Quality Assurance Plan Performance Testing – Tool Selection It is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.34 Qtest Qtest is automated load testing and monitoring solution, offering high capacity load simulation, extensive monitoring capability. Qtest supports all Web, Web Service, J2EE, .Net, ASP, AJAX, CGI, and Mainframe Portal environments and, in particular, complex Web applications like Siebel, SAP, and Epiphany. With its non-protocol based Winload module, Qtest also supports any non-web application. The tool offers sophisticated and complex frameworks, but nowhere does it provide Silverlight or Smart Client based on WCF. It is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.35 HammerHead 2 Hammerhead 2 is a stress testing tool designed to test out your web server by initiating multiple connections from IP aliases and simulated numerous (256+) users at any given time. The rate at which Hammerhead 2 attempts to pound web server site is fully configurable. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.36 Hammerora Hammerora is a load generation tool for the Oracle Database and Web Applications. Hammerora includes pre-built schema creation and load tests based on the industry standard TPC-C and TPC-H benchmarks to deploy against the Oracle database with multiple users. Hammerora also converts and replays Oracle trace files and enables Web-tier testing to build bespoke load tests for your entire Oracle application environment. The tool provides no mechanism to test web services or RIA based or desktop based smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.37 Pylot Pylot is open performance and scalability testing tool, capable of running HTTP load tests to compliment capacity planning, benchmarking, analysis, and system tuning. Pylot generates concurrent load (HTTP Requests), verifies server responses, and produces reports with metrics. Tests suites are executed and monitored from a GUI. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.38 OpenWebLoad preVue-X provides non-intrusive regression and performance testing of X-Window applications. preVue-X accurately emulates multiple users while requiring no special hooks into your application Copyrights © NetSol Technologies Ltd. All rights reserved 10 Unauthorized copy or use of this document is strictly prohibited
  • 17. Software Quality Assurance Plan Performance Testing – Tool Selection or X Libraries, thus guaranteeing that the application tested is the same application that will be deployed. preVue-X operates at the X protocol level and independently of the GUI. It automatically collects all user interactions with the application (e.g., mouse movements, button activity, keystrokes, think time delays) into an easily editable test script for playback. Features include automated synchronization and time-stamping; geometry independence and script viewer The tool does not speak about WCF in specific or smart clients. Need further RND to reach a conclusion. 2.39 SD’s Profiler SD's Profiler collects information about execution, frequency of all blocks of code in an application. A graphical display sorts this information by file or by frequency, and the relative execution costs are shown graphically superimposed over the source code. Available for C, COBOL, and Java. For C and Java, easily customized to work in embedded or real-time contexts. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.40 Benerator benerator is a framework for creating realistic and valid high-volume test data, used for (unit/integration/load) testing. Metadata constraints are imported from systems and/or configuration files. Data can be imported from and exported to files and systems, anonymized or generated from scratch. Domain packages provide reusable generators for creating domain- specific data as names and addresses internationalizable in language and region. It is customizable with plugins and configuration options. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.41 SOATest (by Parasoft) Parasoft SOA Quality Solution continuously measure how each service conforms to the often dynamic expectations defined by both your own organization and your partners, reduces the complexity of testing in today’s heterogeneous environment – with limited visibility/control of distributed components or vendor-specific technologies. It validates all critical aspects of complex transactions, which may extend beyond the message layer through a web interface, ESBs, etc. The tool provides no clarity in order to test WCF or smart client or web services. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.42 Scapa Test and Performance Platform Scapa Test and Performance Platform is testing, diagnosis and monitoring product. The Platform works by simulating multiple users of a computer system. By measuring the performance seen by virtual users, it is possible to understand how the system would behave under a corresponding level of real usage. Copyrights © NetSol Technologies Ltd. All rights reserved 11 Unauthorized copy or use of this document is strictly prohibited
  • 18. Software Quality Assurance Plan Performance Testing – Tool Selection The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.43 CLIF Load Injection CLIF is a modular and flexible distributed load testing platform. It may address target systems reachable from a Java program (HTTP, DNS, TCP/IP...) CLIF provides 3 user interfaces (Swing or Eclipse GUI, command line) to deploy, control and monitor a set of distributed load injectors and resource consumption probes (CPU, memory...) An Eclipse wizard helps programming support for new protocols. Load scenarios are defined through XML-editing, using a GUI, or using a capture tool. The scenario execution engine allows the execution of up to millions of virtual users per load injector. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.44 ContiPerf ContiPerf is a small testing utility that enables the user to easily leverage JUnit 4 test cases as performance tests e.g. for continuous performance testing. It is inspired by JUnit 4's easy test configuration with annotations and by JUnitPerf's idea of wrapping Unit tests for performance testing, but more powerful and easier to use. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.45 Coveriety TBC The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.46 D-ITG (Distributed Internet Traffic Generator) D-ITG (Distributed Internet Traffic Generator) is a platform capable to produce traffic at packet level accurately replicating appropriate stochastic processes for both IDT (Inter Departure Time) and PS (Packet Size) random variables. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. Copyrights © NetSol Technologies Ltd. All rights reserved 12 Unauthorized copy or use of this document is strictly prohibited
  • 19. Software Quality Assurance Plan Performance Testing – Tool Selection 2.47 Database Opensource Test Suit (DOTS) The Database Opensource Test Suite (DOTS) is a set of test cases designed for the purpose of stress- testing database server systems in order to measure database server performance and reliability. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.48 DB Monster DBMonster is an application to generate random data for testing SQL database driven applications under heavy load. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.49 Deluge Deluge is an open-source web site stress test tool. It simulates multiple user types and counts, includes proxy server for recording playback scripts, and log evaluator for generating result statistics. This tool has not been updated since 2002. This tool is no longer under active development although it is still available on Sourceforge. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.50 DieselTest Dieseltest is a Windows application that simulates hundreds or thousands of users hitting a website. BEWARE: This tool has not been updated since 2001. It remains listed here in case anybody wishes to take it over. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.51 Faban Faban is a facility for developing and running benchmarks, developed by Sun. It has two major components, the Faban harness and the Faban driver framework. The Faban harness is a harness to automate running of server benchmarks as well as a container to host benchmarks allowing new benchmarks to be deployed in a rapid manner. Faban provides a web interface to launch & queue runs, and extensive functionality to view, compare and extract graph of outputs. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. Copyrights © NetSol Technologies Ltd. All rights reserved 13 Unauthorized copy or use of this document is strictly prohibited
  • 20. Software Quality Assurance Plan Performance Testing – Tool Selection The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.52 FunkLoad FunkLoad is a functional and web performance testing tool, written in Python. It can be used for functional and regression testing of web projects as well as performance testing and stress testing to overwhelm the web application resources and test the application recoverability. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.53 Curl-Loader Written in C, Curl-Loader is a web application testing and load generating tool. Provides an open- source alternative to Spirent Avalanche and IXIA IxLoad. The loader uses real HTTP, FTP and TLS/SSL protocol stacks, simulating tens of thousand and hundred users/clients each with own IP-address. The tool supports user authentication, login and a range of statistics. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.54 Spirant Avalanche TBC. 2.55 IXIA IxLoad TBC. 2.56 Paessler TBC The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.57 AQ TestComplete and AQ Time AQTime is a performance profiler and memory allocation debugger for 32-bit and 64-bit Windows and .NET-connected applications. AQtime can be integrated into Microsoft Visual Studio, Borland Developer Studio or CodeGear RAD Studio or CodeGear RAD Studio or it can run as a stand-alone application. AQtime can also determine what caused bottlenecks. The tool provides no mechanism to test RIA based applications or WCF calls. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. Copyrights © NetSol Technologies Ltd. All rights reserved 14 Unauthorized copy or use of this document is strictly prohibited
  • 21. Software Quality Assurance Plan Performance Testing – Tool Selection 2.58 Test Maker TestMaker offers building and running intelligent test agents that test Web-enabled applications for scalability, functionality, and performance. It works with object-oriented scripting language (Jython) to build intelligent test agents, an extensible library of protocol handlers (HTTP, HTTPS, SOAP, XML-RPC, SMTP, POP3, IMAP), a new agent wizard featuring an Agent Recorder to write scripts, a library of fully-functional sample test agents, and shell scripts to run test agents from the command line and from unit test utilities. The tool provides no mechanism to test RIA and smart client. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.59 TP Test TPTEST measures the throughput speed to and from various reference servers on the Internet. The use of TPTEST may help increase the consumer/end user knowledge of how Internet services work. The tool is very basic and cannot be graded as performance testing tool for applications. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.60 Tsung Tsung is a distributed load testing tool. It is protocol-independent and can currently be used to stress HTTP, SOAP and Jabber servers (SSL is supported). It simulates complex user's behavior using an XML description file, reports many measurements in real time (including response times, CPU and memory usage from servers, customized transactions, etc.). HTML reports (with graphics) can be generated during the load. For HTTP, it supports 1.0 and 1.1, has a proxy mode to record sessions, supports GET and POST methods, Cookies, and Basic WWW-authentication. It has already been used to simulate thousands of virtual users. The tool requires further RND to experiment with OUR PRODUCT POS. 2.61 Tnsgen Tnsgen is test data generation tool which operates under Solaris, Linux and Windows, capable of generating thousands of records per second in XML, SQL or user-defined formats. Easily extensible to include project-specific data sets. The tool is a data generating tool, it cannot be graded as performance testing tool. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.62 N-Time N-Time tool is very similar to NUnit tool to perform repeatable tasks that help managers, architects, developers and testers to test an application against its performance. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. Copyrights © NetSol Technologies Ltd. All rights reserved 15 Unauthorized copy or use of this document is strictly prohibited
  • 22. Software Quality Assurance Plan Performance Testing – Tool Selection 2.63 p-unit p-unit is an open source framework for unit test and performance benchmark, under GPL license. p-unit supports to run the same tests with single thread or multi-threads, tracks memory and time consumption, and generates the result in the form of plain text, image or pdf file. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.64 Lobo Lobo tool supports performance testing of Java based applications only. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.65 MessAdmin MessAdmin is a light-weight and non-intrusive notification system and HttpSession administration for J2EE Web Applications, giving detailed statistics and informations on the application. It installs as a plug-in to any Java EE WebApp, and requires zero-code modification. The tool is specific for java based application. It provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.66 Multi Router Traffic Grapher (MRTR) Multi Router Traffic Grapher is written in perl and works on Unix/Linux as well as Windows and even Netware systems. MRTG is free software licensed under the Gnu GPL. The tool is specific for java based application. It provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.67 RRRD Tool RRRD Tool is data logging and graphing system for time series data. It can be used to write custom monitoring shell scripts or create whole applications using its Perl, Python, Ruby, TCL or PHP bindings. The tool is barely into application performance testing. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.68 Multi-Mechanize Multi-Mechanize is an open source framework for performance testing. It allows running simultaneous python scripts to generate load against a web site or web service. Copyrights © NetSol Technologies Ltd. All rights reserved 16 Unauthorized copy or use of this document is strictly prohibited
  • 23. Software Quality Assurance Plan Performance Testing – Tool Selection The tool does not address modern needs of RIA based clients and smart clients. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.69 Stress Driver Stress Driver is a general-purpose stress test tool. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.70 Load Impact LoadImpact is online load testing service from loadimpact.com for load- and stress- testing; with access to distributed network of load generator nodes – server clusters with very fast connections to enable simulation of simultaneous users accessing your website concurrently. Free low level load tests for 1-50 simulated users; higher levels have monthly fees. No claim to test RIA client on their site. No support for smart client. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.71 Pure Load PureLoad is a load testing tool that simulates concurrent users executing requests against server based applications. Use PureLoad to verify performance and stability of your server applications. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.72 Pure Agent PureAgent is a performance monitoring tool that replicates real users generating activities on server applications. PureAgent measures the total round trip time it takes to execute one or many user operations accessing either static or dynamic content. The results are presented as charts in a standard web browser, alarm levels can be defined which might trigger notifications to be sent by Email. The monitored scenario of user operations is created using the PureTest application. The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.73 FWPTT fwptt is used for load testing of web applications only. It can record normal and ajax requests. It can understand HTML being returned from server, hence, it works with asp.net, jsp, php or other such applications. The tool recently supported AJAX as well. Since AJAX is the extent of the tool, the tool is too basic to meet the complex requirements of OUR PRODUCT where web is using silverlight and smart client. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. Copyrights © NetSol Technologies Ltd. All rights reserved 17 Unauthorized copy or use of this document is strictly prohibited
  • 24. Software Quality Assurance Plan Performance Testing – Tool Selection 2.74 JBlits Pro JBlits gives continuous heavy loading and automated response validation. Easy to use, quick to setup. Fully featured with logging, graphs, stats, scheduling, HTTP debugger etc. Full Java API. Works in Java, works for Java. The tool is not meant for Silverlight and smart client. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.75 JCrawler JCrawler can start with a set of starting URLs and it will begin crawling from that point onwards, going through any URLs it can find on its way and generating load on the web application. Although, the load parameters (hits/sec) are configurable, JCrawler doesn’t look like a real performance testing tool. There are no built-in mechanism for correlation and parameterization. The tool, at best, can simulate load on static websites where no custom data is required. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.76 PandoraFMS Pandora FMS is a monitoring Open Source software and watches systems and applications. Pandora FMS could detect a network interface down, defacement in website, a memory leak in server application, or the movement of any value of the NASDAQ new technology market. Pandora FMS could send out SMS message when system fails... or when Google's value drop below US$ 500. Although PandoraFMS provides features unique to itself and it can be very useful in many ways, PandoraFMS fails to simulate load on RIA and Smart Client. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.77 Siege Siege can be used to test over HTTP and HTTPS. SIEGE is an http regression testing and benchmarking utility, designed to measure the performance of code. It lets the user hit a webserver with a configurable number of concurrent simulated users. SCOUT surveys a webserver and prepares the urls.txt file for a siege. In order to perform regression testing, siege loads URLs from a file and runs through them sequentially or randomly. Scout makes the process of populating that file easier. You should send out the scout, before you lay siege. With the above limitation of the tool, Siege is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.78 HTTP Load http_load is a multiprocessing http test client. As the name suggests, the tool works specifically for HTTP protocol. This helps cater plain HTML applications only. http_load runs multiple http fetches in parallel, to test the throughput of a web server in a single process. It can be configured to do https fetches as well. It processes a file containing a list of URLs that may be fetched, a flag specifying how to start connections (either by rate or by number of simulated users), and a flag specifying when to quit (either after a given number of fetches or a Copyrights © NetSol Technologies Ltd. All rights reserved 18 Unauthorized copy or use of this document is strictly prohibited
  • 25. Software Quality Assurance Plan Performance Testing – Tool Selection given elapsed time). There are also optional flags for checksums, throttling, random jitter, and progress reports. With all above, the tool serves very basic level requirement for OUR PRODUCT. Provides no mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.79 httPerf httperf provides facility for generating HTTP workloads. The three distinguishing characteristics of httperf are: • its robustness, which includes the ability to generate and sustain server overload, • support for the HTTP/1.1 and SSL protocols, and • its extensibility. With all above, the tool serves very basic level requirement for OUR PRODUCT. Provides no mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.80 iPerf iperf was developed as a modern alternative for measuring maximum TCP and UDP bandwidth performance. iperf allows the tuning of various parameters and UDP characteristics. iperf reports bandwidth, delay jitter, datagram loss. With all above, the tool serves very basic level requirement for OUR PRODUCT. Provides no mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.81 Site Blaster Site Blaster is a web site load and stress testing tool. It can be used to rapidly submit requests to a site. Or, it can pause a random amount of time between submissions; approximating the behavior of a user. While the testing is being performed, the pages being tested will be displayed. When testing is complete, a report is available that can be viewed or printed. It provides no mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.82 RawLoadTester RLT calls the selected URL multiple times and tells server response time. It writes some additional runtime details to the PHP log file to optionally do more granular analysis afterwards. Although the server processes most of the statistics, all URL requests come from the browser. You can run as many browsers and workstations simultaneously as you want. Copyrights © NetSol Technologies Ltd. All rights reserved 19 Unauthorized copy or use of this document is strictly prohibited
  • 26. Software Quality Assurance Plan Performance Testing – Tool Selection RLT provides no mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.83 Allmon Allmon is an open source, generic system collecting and storing metrics used for performance and availability monitoring. It provides no mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.84 AppsWatch Appswatch is an automated testing tool for performance, and GUI testing and SLA monitoring and alerting. It is protocol independent and works with any application accessible from windows desktop. It works for Citrix, Java, .NET, Oracle, SAP, Eclipsys, Mckessen, Meditech, GE Healthcare, Epic, Cerner, Client-Server, Flex, Flash and even custom applications It provides no mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.85 BurnIn Test BurnInTest allows sub-systems of a computer to be simultaneously stress tested for endurance, reliability and stability. BurnInTest tests the CPU, hard drives, CD ROMs, DVDs, sound cards, 2D graphics, 3D graphics, RAM, network connections & printers. The Pro version can also test tape drives, USB, serial and parallel ports. This can all be done simultaneously as the application is multithreaded. It provides no mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.86 Capacity Calibration (CapCal) CapCal writes code in Java for measuring web site performance under various loads--by actually generating those loads, in real time, from computers scattered all across the Internet. CapCal can also be distributed across your LAN and WAN, giving realistic performance data that matches end user's experience. It provides no mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. Copyrights © NetSol Technologies Ltd. All rights reserved 20 Unauthorized copy or use of this document is strictly prohibited
  • 27. Software Quality Assurance Plan Performance Testing – Tool Selection 2.87 Chariot Chariot tests and troubleshoots any segment of network and provides comprehensive reports of the results by emulating transaction traffic from real applications. It can be used to predict how well network infrastructure will handle popular applications prior to deployment, and to isolate and evaluate the performance of new network devices. Chariot is highly scalable and can be used to set up and control tests—emulating up to 10,000 connections and hundreds of thousands of end- users—all from a central console. It cannot be graded as real performance testing tool since it aims for network and infrastructure against internet applications. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.88 CitraTestVU CitraTestVU is a load testing tools, works in Visual Basic scripting to simulate Load on Citrix & Microsoft Terminal Services Hosted Environments. The tool is focused on Java only. It provides no mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.89 SSW Performance Pro SSW Performance PRO! isolates performance bottle-necks in your Access 97, 2000, 2002 or 2003 app. It checks every form, combo and list box, and reviews SQL statements. It reports exactly what to fix to improve the speed of your application. It’s especially useful after you've upsized an application to SQL Server to pinpoint black spots in your app. The tool is very basic and mainly targets MS Access. It cannot be graded a commercial level tool. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.90 e-Load e-Load is enables accurate testing of the response times and scalability of web applications and web services. e-Load can concurrent users, executing real business transactions to analyze how well web software applications will perform under load. The tool provides no mechanism to test RIA based and desktop based applications. It supports Web Services but not WCF. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.91 MLoad MLoad generates high levels of realistic network and user traffic ensuring that WAP Gateways, MMSCs and web based mobile infrastructure elements meet the high-bandwidth, high- performance demands of a mobile internet service. The toolset includes a test harness to allow acceptance testing of mobile elements against performance requirements in the deployed environment. The tool focused on data flux and network bandwidth only. It provides no mechanism to test RIA based and desktop based applications. It supports Web Services but not WCF. Copyrights © NetSol Technologies Ltd. All rights reserved 21 Unauthorized copy or use of this document is strictly prohibited
  • 28. Software Quality Assurance Plan Performance Testing – Tool Selection The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.92 Monitor Master Monitor Master is a quality management solution for wireless data services - WML/XHTML, SMS and MMS. Monitor Master helps Operators and Content Developers ensure that end-users of wireless applications will have the best quality of experience possible - regardless of the handset being used. The tool focuses wireless protocols and not RIA based and desktop based applications. It supports Web Services but not WCF. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.93 Forecast Forecast is a easily and accurately tests the performance and scalability of IT applications and systems. No support for WPF, WCF or Smart Client. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.94 GS DataGenerator GS DataGenerator generates high volumes of 'intelligent' test data on virtually any relational database, creates business trends and patterns, simulates business workflow, mimics potential data errors Employ number of predefined functions. This broad range of functionality allows testing database performance and functionality during development. It’s very basic tool, works in DOS and provides no support for WPF, WCF or Smart Client. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.95 vPerformer vPerformer is used to assess the performance and scalability of web applications. vPerformer allows assesses how web application responds when it is concurrently accessed by a large number of users. It can measure the performance characteristics of application by recording and replaying automated scripts that simulate a large number of concurrent virtual users. The tool is very basic and doesn’t address modern needs like Silverlight etc. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.96 JUnit JUnitPerf is a collection of JUnit test decorators used to measure the performance and scalability of functionality contained within existing JUnit tests The tool is specific for Java related apps. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. Copyrights © NetSol Technologies Ltd. All rights reserved 22 Unauthorized copy or use of this document is strictly prohibited
  • 29. Software Quality Assurance Plan Performance Testing – Tool Selection 2.97 Xceptance LoadTest Xceptance LoadTest for web and Java and other apps includes recording capabilities. XLT Cloud Service available. Tests implemented as JUnit 4 test cases. For web-based tests, the framework provides a (headless) browser that can emulate Internet Explorer or Firefox behavior. Can execute client-side JavaScript in the emulated web browsers and that way it simplifies the creation of test cases for Web 2.0 applications. It is platform independent since the tool is implemented in Java; test scripting in Java or Ruby. The tool is free for up to five virtual users. The tool is focused on Java only. It provides no mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.98 zapwireless zapwireless is a wireless throughput test tool which can be used to fully characterize the statistical performance of a wireless link. It provides no mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.99 Seagull Seagull was primary aimed at IMS protocols, now it is a traffic generator for functional, load, endurance, stress and performance tests for almost any kind of protocol. Currently supports Diameter, XCAP over HTTP, TCAP (GSM Camel, MAP, Win) protocols. It provides no mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.100 SIPp SIPp is a performance testing tool for the SIP protocol. Its main features are basic SIPStone scenarios, TCP/UDP transport, customizable (xml based) scenarios, dynamic adjustement of call- rate and a comprehensive set of real-time statistics. It can also generate media (RTP) traffic for audio and video calls. Although Sipp is a rich tool and may be useful for specific purposes, it provides no mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.101 SLAMD SLAMD Distributed Load Generation Engine is a Java-based application designed for stress testing and performance analysis of network-based applications. SLAMD does not provide any mechanism to test RIA based and desktop based applications – this in turns disregards smart client applications as well. Copyrights © NetSol Technologies Ltd. All rights reserved 23 Unauthorized copy or use of this document is strictly prohibited
  • 30. Software Quality Assurance Plan Performance Testing – Tool Selection The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.102 IxoraRMS IxoraRMS is a monitoring tool with great visualization and customization capabilities. But it severely lacks capabilities to support RIA and modern complex architecture. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.103 j-hawk j-hawk is a Java based open source framework which can be incorporated within an application for performance testing. The idea requires to define *module and its method inside the application and register the same with j-Hawk. j-Hawk executes the modules and generates a graphical performance report which can be analyzed to find performance bottleneck of your application. The tool is specific for Java based application which is not the case with OUR PRODUCT. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.104 Wireshark TBC The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.105 Ostinato Ostinato is an open-source, cross-platform packet/traffic generator and analyzer with a friendly GUI. It aims to be "Wireshark in Reverse" and thus become complementary to Wireshark.The tool is specific for Java based application which is not the case with OUR PRODUCT. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.106 Eclipse Test & Performance Tools Platform Project (EPTP) EPTP supplies frameworks and services that allow software developers to build unique test and performance tools, both open source and commercial, that can be easily integrated with the platform and with other tools. Eclipse Test & Performance Tools Platform Project (TPTP) evidently provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 2.107 Web Polygraph Web Polygraph is a benchmarking tool for caching proxies, origin server accelerators, L4/7 switches, content filters, and other Web intermediaries. This tool focused mainly on Light Directory Access Protocol (LDAP) and web applications working on HTTP protocol. The tool appears to be limited in functionalities it offers. The tool is incapable of handling Rich Internet Application (RIA) such as flex or Silverlight. Copyrights © NetSol Technologies Ltd. All rights reserved 24 Unauthorized copy or use of this document is strictly prohibited
  • 31. Software Quality Assurance Plan Performance Testing – Tool Selection The tool provides no mechanism to test desktop based applications – this in turns disregards smart client applications as well. The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase. 3. Tools Selection Criteria Criteria for selection of Automated Performance Testing tool was defined before proceeding with research activities. The criteria broadly address Business and Technical Requirements and Objectives sought from Performance Testing. Below is a highlight of foundation of criteria: 3.1 Business and Technical Requirements 1. It is essential for the application is able to pass at least 1 test case to perform certain business process. Although, it is best to have reasonably stable application before performance testing starts. If there is urgency, the application should be able to work under at least 1 situation/record-set. 2. It is pivotal to finalize performance test objectives before test scripting is begun. Number of users, data volume and other variables must be known beforehand otherwise unnecessary delays may occur. For example, if the intention is to run endurance or longevity test with 1000 users over 4.5M record base, for 24 hours; then test data requirement will be massive as compared to running 100 users for 1 hour. 3. Dedicated test environment and interface systems are identified and requisition (booking) made in advance and certified to be used for testing. The hardware specifications should be documented for web server, application server, db server and any other layer involved. This should be documented for Load Injectors/Generators as well. 4. System components have been identified and deployed over test environment mimicking (or a close match to) production server. 5. Test IDs/passwords and access rights have been identified, requisition made and approval granted for all involved applications. Test IDs must have the correct access rights for testing for application as required by user’s role. 6. Criteria for test data to be used for testing has been drawn, and reviewed by respective parties involved. 7. Test Data has been created in testing database. 8. System infrastructure and connectivity test are performed prior to each test phase to ensure connectivity is enabled. 3.2 Performance Test Objectives In general, the test objective is to ensure the respective applications meet the stated and implied performance requirements of business; as well as, the statistics remain within the SLA agreed with end customer. Copyrights © NetSol Technologies Ltd. All rights reserved 25 Unauthorized copy or use of this document is strictly prohibited
  • 32. Software Quality Assurance Plan Performance Testing – Tool Selection 1. Identify any bottlenecks and slow system performance 2. Ensure application response time is acceptable with data volume of 4.5 Million records 3. Gauge application responsiveness under Stress 4. Validate application behavior under endurance and longevity (24+ hours) 5. Validate the behavior of Load balancing over cluster nodes equally 6. To ensure that application is functioning properly under load 7. Establish performance benchmarks 8. Validate transactions response time against customers’ SLA 9. Perform root cause analysis for any problems and errors appearing under load – and suggest possible optimization. 10. Draw projection as measures for future needs 11. Ensure smooth implementations with no/minimum disruption to business going live, and applications rollout with minimum risk. 12. Project implementation compliance to ChromeIS processes. 13. Project quality that meets entry and exit criteria for each performance testing phase. There are specific goals of performance testing, like: 1. Measure and benchmark response time for each page/screen to open 2. Measure and benchmark response time against each transaction, like, how long it takes when, saving a proposal, saving a contract, converting a proposal into contract, calculating exposure of a customer, asset is attached to a proposal, rentals are generated, financial analysis is done, early termination is performed etc. 3. Measure and benchmark response time for each service as an individual. 4. This section specifies the test types (with definition for each) to run. This also explains when application will be considered fit to undergo each type of testing and when a particular 5. Measure and benchmark maximum number of users a transaction/service can entertain disregarding response time (peak point identification) 6. Benchmark maximum number of users achieved against test environment (hardware/software specifications) 7. Benchmark maximum hardware utilization against desired number of users (against test environment specifications) 8. Benchmark system behavior against cold startup and warm startup 9. The spike ratio that the system must be capable of handling 10. The peak ratio that the system must be capable of handling? 11. Memory leakage against each business process or individual service Copyrights © NetSol Technologies Ltd. All rights reserved 26 Unauthorized copy or use of this document is strictly prohibited
  • 33. Software Quality Assurance Plan Performance Testing – Tool Selection 12. Identify which SPs fail and at what circumstances, at what line, under what conditions. 13. Facilitate in ensuring load is distributed evenly over all nodes of clusters (as configured) 14. Facilitate ensure seamless mitigation of failover under high load 15. Tolerance ratio: Imposed load ? 25 %? 16. Spike ratio: Imposed load x 3? 17. Burstiness ratio: Imposed load x 5? 18. Increase the load by multiplying the load baseline by 1x, 2x, 3x, 4x, Nx gradually until unacceptable response time is reached through optimization. 19. Pass recommendation in achieving optimal configuration, or identifying target component where optimization could be required. 0.1 Second 10 Second 20 Second 30 Second Acceptable User Experience Unacceptable User Experience Questionable User Experience Business is Closed Response Time Load 60 Second Figure 1, Performance Acceptance at a glance The above assumes, all stakeholders are on same page regarding: • What is transaction response time and what is acceptable response time? • The difference between simultaneous and concurrent users • Which metrics should be collected during all types of performance testing? • What is the correlation between demand and increased load? Copyrights © NetSol Technologies Ltd. All rights reserved 27 Unauthorized copy or use of this document is strictly prohibited
  • 34. Software Quality Assurance Plan Performance Testing – Tool Selection • How do we determine which components are problematic? 3.3 Performance Test Execution Performance testing usually follows a linear path of events: • Create performance-testing scripts and scenarios. • Create Task Distribution Diagram to elaborate maximum number of users over a component as well as maximum number of users accessing system at any point of time. • Execute performance tests as standalone tests to detect issues within a particular business workflow. Repeat execution multiple times to surface most errors, identify root cause of failure. • Execute performance scenarios as a package (i.e. with all BPs together) to simulate real life activities that are measured against performance success criteria. • Monitor all variables involved in Performance Testing. This includes all servers and their components such as processor, memory, NIC, clusters, caching, heap, top SQL queries, SPs, ISS caching, IIS sessions, Kernel, Number of processes, Number of threads, Throughput, DB Sessions, Parameters, Data Buffer Cache Hit Ratio, Shared Pool Hit Ratio, Query to Parse ratio, PGA Usage, I/O read wait time, I/O write wait time, CPU wait time, configurations, hits per second, timeouts etc. These statistics will be collected for Web Server, App Server, DB Server and any other server involved, as well as load injectors. • Collate all statistics and observations. Perform analysis and document recommendations/suggestions. Compile a concise performance testing results report. • Tune the system. • Repeat testing as required. 3.4 Load Description In the context of Performance Testing, “Load” can be broadly classified into 2 categories, namely, Server Based Load and User Based Load. Below is the elicitation of both terminologies. Server-based Load • Number of users and/or sessions • Average session time • Number of page views • Average page views per session • Peak period (e.g., 75% of traffic is from 11:00 AM-4:00 PM) • Number of hits • Average page size • Most requested pages • Average time spend on page Copyrights © NetSol Technologies Ltd. All rights reserved 28 Unauthorized copy or use of this document is strictly prohibited
  • 35. Software Quality Assurance Plan Performance Testing – Tool Selection • Frequency of visits (e.g., 75% of users made one visit) • Demographics • Client information such as browser, browser version, Java script support, Java script enable/disable, and so on. User Based Load: • Number of users • Session length • User activities and frequency of activities per session • Think/Read/Data-input time • Percentage by functional group • Percentage by human speed • Percentage by domain expertise/ familiarity (speed) • Percentage by demographics (arrival rates) Other questions to consider when describing system Load: • What is the definition of “workload”? • How do we size the workload? • What is the expected workload? • What’s the mix ratio of static pages vs. code? • What is the definition of “increased load”? • What is future growth? Can it be quantified? • What is the definition of scalability? A requirements document is filled up for each testing cycles to further clarify performance test requirements. Refer to below example file. 4. Conclusion After assessing large number of tools available in market, we can conclude that HP LoadRunner, IBM Rational Performance Center and JMeter (the top 3 market leaders) do not directly support testing the complex architecture we’re building; at least not as yet. A number of open source tools, freeware utilities, ASM based performance testing services and other freeware utilities do not address the performance testing challenges we‘re facing. JMeter could be a possible option if encryption can be removed from OUR PRODUCT for testing purposes. EggPlant Load Tester described in detail how it can test web services written in .net. This requires further RND and experimentation if time allows. Tsung is an open source framework which claims testing web services and other complex niche technology frameworks. This requires further RND and experimentation if time allows. We’ve had trial runs with Visual Studio and optimistically, we can hope to meet our objectives with MSVS. This is further described in Section 4.1 below. Copyrights © NetSol Technologies Ltd. All rights reserved 29 Unauthorized copy or use of this document is strictly prohibited
  • 36. Software Quality Assurance Plan Performance Testing – Tool Selection 4.1 Selected Tools The testing tool chosen is Microsoft Visual Studio 2013. NeoLoad can be used to identify the sequence of calls made to web services. Although, MSVS this is not specifically designed performance testing in mind, however, this has been picked as a workaround for current situation. Visual Studio offers scripting capabilities, thus enabling to simulate asynchronous multithreaded simultaneous users to the application. This is the first stepping stone for us. Once the load is simulated closed to real behavior, various other tools will be used to identify application behavior and other environmental factors undergoing changes (as described in Section 3) Below table enlists tools which will be used for Performance Testing as well as other activities (e.g. resources monitoring) during performance testing phase: Tool Purpose Visual Studio 2013 For Performance Testing of areas which can’t be worked out with HP PC / LR. HP LoadRunner 12 Automated performance and stress testing NeoLoad Trace client/server communication SolarWind Defects logging and tracking Spotlight Enterprise Resource Monitoring Windows Profiler Extensive Resource Monitoring Table 2. Selected tools for Performance Testing phases 4.2 SWOT Analysis In concluding this research, we’ve learned about numerous testing tools, what those are capable of and their approach towards load simulation. HP LoadRunner is a market leader in performance testing tools, with over 90% market share globally. ChromeIS has mature strong expertise in HP LoadRunner and HP Performance Center and that had always been strength of ChromeIS to address any performance concerns in timely fashion. Unfortunately, HP LoadRunner has very limited support for .net protocol and it induces a lot of errors whilst preparing performance test scripts for OUR PRODUCT. Experiments show that the tool fails in deserialization, decoding and parsing already desterilized fragments of scripts. We can conclude that LoadRunner 12 does not offer what it takes to test a sophisticated .net based application with complex architecture and modern clients. Time has changed and ChromeIS has taken initiatives to re-invent the products using state of the art technologies making Rich Internet Applications and optimizing performance using Smart Client. While this sounds exciting, it brings unforeseen challenges in many ways. Performance and efficiency of software is undoubtedly among most critical aspects of software and this cannot be ascertained without undergoing aggressive performance test. Hence, a workaround has been suggested and Microsoft Visual Studio 2013 will be used for conducting performance testing. Copyrights © NetSol Technologies Ltd. All rights reserved 30 Unauthorized copy or use of this document is strictly prohibited
  • 37. Software Quality Assurance Plan Performance Testing – Tool Selection 4.3 Training Needs ChromeIS has mature expertise in HP LoadRunner and HP Performance Center; however, these tools are out of option. Since ChromeIS will be using Visual Studio 2013 for development of Performance test agents, certain level of resource training is required. A team size of 4-6 dedicated resources is expected to be working for Performance Testing. This team will be briefly trained by QAM on performance testing goals, objectives and overall approach towards meeting quantitative goals. While the conceptual and quantitative goals remain the same, working with the tool itself will be crucial to achieve test objectives. Training programs to be provided to the team to enable successful planning and execution. A training plan can be published separately or as an addendum to this document. 4.4 Lessons Learnt Selection of tool and technology for carrying out performance testing is of utmost significance. The same applies to overall software development as well. We’ve seen in past as well that development technology was picked without consensus, or even consultation with Performance Testing team. Now we’ve understood that performance testing an application involves sophisticated activities, induces unforeseen technical challenges, and imposes technological constraints, thus essentially making it a challenge to workout performance testing. This cannot be simply compared with business testing where there is no or little involvement of technological constrains. It is suggested to keep performance testing team in due consultation when making technology selection or making technical changes to existing application (upgrade of Silverlight version, bringing parts of application logic to thick client). Although the final decision of picking the development technology or modifying underlying components will be under sole discretion of the PM, this discussion will ensure any possible constraints are communicated and any possible risks are addressed 5. Definition of Terms, Acronyms And Abbreviations Term Description DEV Development Team PT Performance Testing SQA Software Quality Assurance Copyrights © NetSol Technologies Ltd. All rights reserved 31 Unauthorized copy or use of this document is strictly prohibited