Getting start with Performance Testing

ALL RIGHTS RESERVED 2013
Agenda:
Introduction to Performance Testing
Why Performance Testing?
Performance Testing jargons
Performance Test Metrics and Process
Limitations of Manual Performance Testing
Benefits of Performance Test Automation
 Tools used for Performance Testing

Performance Testing
Wikipedia says:
“In the computer industry, software performance testing is
used to determine the speed or effectiveness of a computer,
network, software program or device. This process can
involve quantitative tests done in a lab, such as measuring
the response time or the number of MIPS (millions of
instructions per second) at which a system functions.
Qualitative attributes such as reliability, scalability and
interoperability may also be evaluated. Performance testing
is often done in conjunction with stress testing.”
Our Simplified Definitions
Performance Testing = how fast &
stable is the system?

Why Performance Testing?

Oh it’s 10 o’clock
Got to book my Rail
Ticket..

Poor Performance Affects:
• Loss of revenue
 Amazon states that for every 100ms of latency, they
lose 1% of their sales.
 Google's experiments prove statistically that slowing
its search results by 400ms costs millions of dollars
per year due to lost advertisement revenue.
 Industry experts estimate that revenue of $3 billion
has lost in 2012 because of slow site performance.
• Loss of customers
 Google says they could lose 8 million searches daily if
their search results are 1/2 second slower.
 Flipkart says “57% of shoppers will abandon a site
after waiting 3 seconds for a page to load, out of them
80% don’t return again on same site.”

Continued…
• Loss of productivity
 Reddit, states that their performance focus is on page load speed.
“If we can get 10% more performance, we immediately see 10%
more traffic.”
 Better performing website (speed improvements) increased
productivity by 7-12%.
• Backlog of Work
 On an average 1.2 million people try to book tickets in the first ten
minutes of the Tatkal timeframe, Of that only 50000 people come
out of the website with tickets, at the success rate of 4.16% only
due to service backlog during peak.
• Media attention and damage to your brand
 Times Of India Ranked IRCTC “1st” out of 8 ‘worst’ Indian
government websites in 2012.

Why Performance Testing?
• Identifies problems early on before they
become costly to resolve.
• Produces better quality, more scalable code.
• Prevents revenue and credibility loss due to
poor Web site performance.
• Enables intelligent planning for future
expansion.
• To ensure that the system meets
performance expectations such as response
time, throughput etc.

Performance Testing Jargons
 Before going into the details, we should understand the factors that governs
Performance testing:
 Business Transaction
 Think Time
 Concurrent User load
 Simultaneous User load
 Throughput
 Response Time
 Tuning
 Baseline Test
 Benchmarking
 User Abandonment
 Performance Bottleneck
 Load Testing
 Stress Testing
 Spike Testing
 Volume Testing

Business Transaction
• It refers to a sequence of
request-response and the
related information exchange
between the client and the
server in order to accomplish a
business requirement.
• For example, an online banking
customer transferring money
from one account to other is a
business transaction.

Think Time
• It refers to the time taken by the user
for thinking or clicking on any of the
web page links, buttons, etc while
navigating through the web site.
• Think time is a very important
parameter which needs to be set while
scripting the scenarios using the
performance testing tools.
• The business analysts of the
application or web site management
team or sometimes even the end user
survey might give a realistic picture
about the think time requirements of
a transaction.

User Abandonment
• It refers to the situation wherein
the end users exit from the web
site because of the slow
performance of the site.
• The user abandonment rate
varies from web site to web site.
A low priority web site might
experience a high user
abandonment rate compared to
the payroll web site. Analysis on
the abandonment rate of the
site needs to be done before
concluding on the load on the
site.

Simultaneous User load
• The simultaneous users have the
active session on the server at any
point of time wherein each user will be
executing different transactions.
• For example, if we say 100
simultaneous users load, then there
will be 100 active sessions opened up
in the server, wherein each user will be
performing different set of
transactions – one logging in, another
viewing reports, another navigating to
the next page, etc.
• The simultaneous user load of a web
site would be always greater than the
concurrent user load of a web site.

Concurrent User load
• The concurrent users connect to the
server and perform the same
operation at any point of time.
• For example, if we say 100
concurrent user load, all 100 users
would be logging in at the same
point of time, view the reports at the
same point of time, etc. For example,
an online banking web site might
have 10,000 – 20,000 simultaneous
user load, but 1000 to 1500
concurrent user load.

Baseline Test
 Make you familiar with the
operational behavior of
each app/server.
 Tells you all about the
performance of system under
normal conditions.
 It refers to the test conducted to
measure the application
performance for limited virtual
user load.
 The baseline test is often
conducted to collect the metrics
about the system performance for
1,5,50…. user load .

Benchmarking
 Benchmarks are baselines at
known and defined levels of
load.
 The objective of the
benchmark test is to validate
the correctness of the test
scripts and to check the
readiness of the system before
subjecting it to a high load.
 Normally for a benchmark
test, 15 -20% of the target load
can be considered.

It is equally important to find out how much time each of the transactions took to complete.
Response time is defined as the delay between the point of request and the first response from
the product.
The response time increases proportionally to the user load.
Response Time
• It is equally important to
find out how much time
each of the transactions
took to complete.
• Response time is defined
as the delay between the
point of request and the
first response from the
product.
• The response time
increases proportionally
to the user load.

Throughput
• Capability of a product to
handle multiple transactions
in a give period.
• Throughput represents the
number of
requests/business
transactions processed by
the product in a specified
time duration.
• As the number of concurrent
users increase, the
throughput increases almost
linearly with the number of
requests. As there is very
little congestion within the
Application Server system
queues.

Tuning
• Tuning is the procedure by which
product performance is
enhanced by setting different
values to the parameters of the
product, operating system and
other components.
• Tuning improves the product
performance without having to
touch the source code of the
product.

Performance Bottleneck
• It refers to the slow spot, the effects
of which are widely felt. It refers to
the situation/areas which do not
allow the application to perform as
per its ideal specifications.
• For example, the response time
increase for the load of 100 virtual
users because of improper setting of
HTTP connections parameter in the
IIS server, CPU utilization reaching
95% during 100 users load are
typical performance bottlenecks. A
bottleneck might lead to a failure if
mitigation actions are not taken.

Load Testing
• Process of exercising the system
under test by feeding it the largest
tasks it can operate with.
• Constantly increasing the load on the
system via automated tools to
simulate real time scenario with
virtual users.

Stress Testing
• Trying to break the system under
test by overwhelming its resources
or by taking resources away from
it.
• Purpose is to make sure that the
system fails and recovers
gracefully.

Spike Testing
• It refers to test conducted by
subjecting the system to a short
burst of concurrent load to the
system. This test might be essential
while conducting performance tests
on an auction site wherein a sudden
load is expected.
• The goal is to determine whether
performance will suffer, the system
will fail, or it will be able to handle
dramatic changes in load.

Volume Testing
• It refers to the tests designed to
measure the throughput of the
system more in a batch
processing, messaging kind of
environment. The objective of
this test is to identify the
processing capacity.
• For example, if you want to
volume test your application
with a specific database size, you
will expand your database to
that size and then test the
application's performance on it.

Transaction Based Metrics Server Based Metrics
Metrics to be Captured Comments
Throughput (per Sec) Transactions per sec
Response Times / Elapsed
Times
Time taken to process a
transaction
Types of Errors Totals for different
types of errors for a
particular test
Count of Errors Total errors for a
particular test.
Transaction Count Total transactions
processed for the time
period
Metrics to be Captured Comments
CPU Usage System%
Idle%
Memory Usage Used%
Used in GB
Memory available
Disk I/O Disk Read KB/sec
Disk Write KB/sec
IO/sec
Network Activity MB/sec
Packets/sec
Size of packets
Bandwidth used

Performance Testing Process

Performance Testing
Manual OR Automation…

Testers
Load Generation
System Under Test
Do you have the testing resources?
• Testing personnel
• Client machines
How do you coordinate and synchronize users?
How do you collect and analyze results?
How do you achieve test repeatability?
Analysis?
123.20
All of you,
click the
GO button
again
Manual Performance Testing
Limitations
Web server Database
server
Coordinator

Continued…
Manual Testing Limitations
 Expensive, requiring large amounts of both personnel and machinery.
 Complicated, especially co-ordinating and synchronising multiple testers
 Involves a high degree of organization, especially to record and analyse
results meaningfully
 Repeatability of the manual tests is limited

Load Generation System Under Test
Benefits of Automation
Web server Database
server
Vuser
host
Analysis
Controller
Solves the resource limitations
•Runs many Vusers on a few machines
•Controller manages the virtual users
•Analyze results with graphs and reports
Solves the resource limitations
•Runs many Vusers on a few machines
•Controller manages the virtual users
•Analyze results with graphs and reports

Continued…
Using Automated Tools
 Reduces personnel requirements by replacing human users with virtual users
or Vusers. These Vusers emulate the behaviour of real users.
 Because numerous Vusers can run on a single computer, the tool reduces the
amount of hardware required for testing.
 Monitors the application performance online, enabling you to fine-tune your
system during test execution.
 It automatically records the performance of the application during a test. You
can choose from a wide variety of graphs and reports to view the
performance data.
 Because the tests are fully automated, you can easily repeat them as often as
you need.

Some common Tools used for Performance
Testing
Open Source
• OpenSTA
• Diesel Test
• TestMaker
• Grinder
• LoadSim
• Jmeter
• Rubis
Commercial
• LoadRunner
• Silk Performer
• Qengine
• Empirix e-Load

Getting start with Performance Testing

More Related Content

What's hot

Viewers also liked

Similar to Getting start with Performance Testing

Recently uploaded

Getting start with Performance Testing