1. The document discusses using cloud computing for performance testing by provisioning virtual machines and load generation servers in the cloud instead of on-premise servers. 2. Commercial testing products and open-source frameworks like JMeter can be used for cloud-based performance testing, with benefits including lower costs, ability to simulate large-scale loads, and geographic distribution. 3. A case study describes a custom Hailstorm framework built on JMeter that was able to simulate 40,000 concurrent users for a client, providing rapid and cost-effective performance metrics at scale.

1. D I S C I P L I N E D I N N O VAT I O N
TEST PERFORMANCE USING CLOUD
Abhinav Maheshwari (Chief Scientist & Director, 3Pillar Labs)
Sayantam Dey (Architect, 3Pillar Labs)

2. OBJECTIVES
1. What are the benefits of using the cloud for
testing performance over traditional means?
2. What are the various tools available for
cloud based performance testing?
3. How can we use open source load testing
frameworks in the cloud?
4. How to design a custom framework for rapid
testing with large number of concurrent
users?

3. TRADITIONAL PERFORMANCE TESTING
Load Generation
Server (High end or
multiple servers)
Application Server
Workstation
1. Deploy replica of the
production environment
in the LAN
2. Deploy a high-end
server for generating
load (on the same LAN)
3. Run agents of a load
test tool like JMeter,
Grinder or WAPT
4. Connect remotely to
load generation server
5. Load generation server
collects metrics and
generates reports

4. CLOUD BASED PERFORMANCE TESTING
Load Generation
Server (High end or
multiple servers)
Application Server
Workstation
Cloud machines
in different
geographies
Application Server
Workstation

5. BENEFITS OF USING THE CLOUD
Traditional Approach Cloud based Approach
Provisioning physical machines is costly,
specially for medium sized companies
Large number of small, virtual machines
running in bursts, ideal usage of cloud
from a cost perspective
Manual provisioning of load generation
server requires time and effort
Simpler provisioning, can even be
automated using API
Application needs to be deployed in the
same LAN
Application may be hosted at any public IP,
also more realistic from latency point of
view
Not dynamic to address any significant
increase in the performance requirement
Distribute load generation over multiple
instances which can be increased
dynamically
Extremely difficult to simulate
geographically distributed load
Cloud clusters in different geographic
locations can simulate geographically
distributed load






6. CLOUD APPROACH – CHOICES ?
Commercial Products
1 Record or upload scripts
for usage scenarios using
the web based interface
2 Configure testing
parameters like number of
virtual users, time for test,
etc. through interface
3 Start the test through web
based interface
4 Download reports and
analyze

7. Open source frameworks
1 Create the required number of VM
instances on the cloud, deploy the chosen
load testing framework
2 Configure all the parameters for the
framework and upload scripts written for
the chosen framework
3 Run agents on the system under test and
then start the test
4 Download reports and analyze
CLOUD APPROACH – CHOICES ?
Commercial Products
1 Record or upload scripts for usage
scenarios using the web based interface
provided by the vendor
2 Configure testing parameters like number
of virtual users, time for test, % allocation
to usage scenarios etc. through interface
3 Start the test through web based interface
4 Download reports and analyze

8. Open source frameworks
• Very popular, large user base
• Very little programming
knowledge required
• Handles many protocols
• Plugin architecture for
extension
• Requires programming
• Scripting in Jython or Clojure
• Distributed framework
CLOUD APPROACH – CHOICES ?
Commercial Products
• JMeter compatible
• Real time reports
• 10K virtual users x 25 hours
@ $1000
Web Performance
• Selenium IDE scripts
• Real browser or simple
request playback (virtual
user)
• 10K virtual users x 25 hours
@ $3750
• Custom scripts
• Real time analytics
• Custom pricing

9. Open source frameworks
Only few frameworks are mature enough
to work over the cloud
Only the cost of cloud infrastructure, which
is much less expensive
Completely flexible as scripting is done
through open APIs
Can only be used through command line,
user has to care about calculating VM
requirement, creation and closure
Reports are basic and sometimes not
helpful
A CUSTOM FRAMEWORK?
Commercial Products
Not many choices since there are only a
few products in the market
May prove to be expensive for repeated
load testing
May use recording or proprietary scripting,
which may not be fully flexible
Provide web-based interface to directly run
tests without worrying about virtual
machines
Provide easy to use reports


 X

Sizable investment - Suitable for organizations needing repeated performance tests

10. CASE STUDY - HAILSTORM
Configuration
 JMeter
 Amazon
 SUT Provisioning Manager
JMeter
Scripts
Script Manager
JMeter Controller
Data Collector
Report Engine
- JFreeChart
AMI
- JMeter
- JRE
Agent
System Under Test (SUT)
Agent
Agent
H
T
T
P

11. CASE STUDY - HAILSTORM
Observations
Sharp rise in response time
for over 50 users
“Detail Page” degrades
significantly for 75 users and
above
System is stable but peaks
out at 75 concurrent users
and starts degrading at 100
users
Maximum throughput of 3.2
transactions per second at 75
users is low
© Copyright 2012 3PILLAR GLOBAL All rights reserved 11
Sample Report Summary

12. CASE STUDY - HAILSTORM
© Copyright 2012 3PILLAR GLOBAL All rights reserved 12
Page Response
time Avg.
(ms)
Response
time 90tile
(ms)
% errors
Home Page 1384 2293 0.0
Login Page 559 764 0.0
Login 1050 1740 0.0
Browse 1996 2898 0.25
Category
Browsing
2553 3601 0.0
Detail Page 5368 7231 0.27
Search 2054 2948 0.0
Sample Report for one iteration

13. CASE STUDY - HAILSTORM
• Application
– A leading provider of online classroom and instruction
solutions
• Key Challenge
– Required simulation of 40,000 concurrent users
– Rapid and cost effective solution to capture load
metrics for large number of concurrent users
• The Solution
– Different JMeter scripts/plans seamlessly integrated
– Engaging reports pointing out bottlenecks generated