Many cloud-based performance and load testing tools claim to offer “cost-effective, flexible, pay-as-you-go pricing.” However, the reality is often neither cost-effective nor flexible. With many vendors, you will be charged whether or not you use the time (not cost effective), and you must pre-schedule test time (not always when you want and not always flexible). In addition, many roadblocks are thrown up—from locked-down environments that make it impossible to load test anything other than straight-forward applications, to firewall, security, and IP spoofing issues. Join Leslie Segal to discover when it makes sense to set up your own cloud-based performance test lab, either as a stand-alone or as a supplement to your current lab. Learn about the differences in licensing tools, running load generators on virtual machines, the real costs, and data about various cloud providers. Take home a road map for setting up your own performance test lab—in less than twenty-four hours.

1. Build Your Own
Performance
Test Lab in the Cloud
Leslie Segal
Testware Associate, Inc.
leslie.segal@testwareinc.com

2. Problem(s) I was trying to Solve
Need more
VUs
Don’t have
Tool
Times up –
reschedule
test
Use Vendor
Cloud
Production Issue
– need to run
test now
Use Vendor
Cloud
Crap! How fast can
I build my own ?
Crap! I should just
build my own
Can’t run test –
need dll
for login
Schedule
Test
Times up –
reschedule
test
Times up –
reschedule
test
Can’t run true
test – can’t
spoof IPs
Can’t run test –
Tool version
different
2

3. Issues with Vendor Cloud
Test Lab
• Must schedule in advance
• Limit on number of Load Generators (LG) you
can use (1 LG per 1,000 vu)
• Can’t install anything on cloud machine
• Permissions typically locked down
• Different version of tool software
• Accessing your application under test
• (open your firewall, open cloud firewall)
3

4. Advantages of Setting up your Own
Cloud-Based Performance Test Lab
• Cost (hardware & tool)
• Need to look at all costs to determine if truly cost
effective
• Time (don't have to wait for long approval
processes for HW & SW)
• Control when (right-now), where (US, CAN, EU)
and how many Load Generators
• Easy way to try vendor tools
• Test tool upgrades
4

5. Why Test from Different
Locations?
• Different Users in different locations will have
different experiences
• Those closest to your servers typically have the best
experience
• Users inside and outside the firewall will have
different experiences
• Using a Content Delivery Network (CDN)?
• Need to test that time sensitive data is consistent
throughout
• Are you mobile?
• Huge variety of mobile networks throughout the world
5

6. Performance Test Architecture
AUT
Local Area
Network
Load
Generators
Controller
Analysis
6

7. Tool Components
HP LoadRunner
Borland SilkPerformer
• Controllers individually
licensed
• Multiple types of Virtual
User protocols
• Have cloud offering (uses
Amazon Cloud)
• Can supplement existing
licenses with cloud licenses
or use standalone
• Need license server
• Separate component for
scripting
• Unlimited controllers
• 3 Levels of Virtual User
Protocols
• Have cloud offering (uses
Amazon Cloud)
• Can supplement existing
licenses with cloud licenses
or use standalone
• Need license server
• Integrated scripting
component
7

8. Load Generators
• Number and size depends on several factors
• Number of virtual users (VU) you want to generate
• Type of VU
• Web service, DB, Web, Mobile
• Type of Application
• Thin client, fat client, amount of network traffic
• Distribution of Load and Transactions
• 50% of normal load from West Coast/20% Canada/30% East
Coast
• 10% of users download content from your site
• 90% of users outside the firewall
8

9. Load Testing on Physical
HW vs. VM
• Cloud machines are typically Virtual Machines
Differences Between Physical & VM
• VM is a slice of a physical machine
• VM needs to perform an extra step to ‘talk’ to actual
hardware
• Excessive requests can skew results
• VM can handle about 80% of load of same VU on
physical machine
• Need to actively monitor CPU, Memory, I/O, Network
Utilization of LG while test is running
9

10. Cloud Cost Components
Machine Size
Storage
10

11. Machine Types
• Options
• On Demand
• Reserved
• Size
•
•
•
•
Standard
High Memory
High CPU
High I/O
• Operating Systems
• Linux
• Windows
• Ubuntu
• DB
• Oracle
• SQL Server
Can start small and resize any instance
11

12. Amazon Cloud Sample Pricing
12

13. Rackspace Pricing (Windows)
13

14. Netel Cloud (CAN $)
14

15. Use Calculators to Estimate
15

16. Calculator - Amazon
Note Large = 7.5GB RAM
16

17. Cost for Data Transfer
Amazon
Rackspace
17

18. Throughput limitations
• Rackspace
18

19. IP Addresses (IPv4)
• Amazon
• Each AWS Cloud Server comes with a non-persistent private IP address
NATed to a public IP address AWS allows 5 Elastic IP addresses per
account
• Cost for not used IP addresses - $0.005 per Elastic IP address not associated
with a running instance per hour on a pro rata basis (~$44/yr)
• Rackspace
• Each Rackspace Cloud Server comes with a dedicated and persistent
public IP address – get same IP if shut down and restart instance
• Note that the server must remain in Active state to retain IP and there is a
charge for an Active Server that is powered off
• Can purchase 5 public IP addresses for $2/month/IP
19

20. Locations (as of Jan 2013)
Amazon
Rackspace
US East (N. Virginia)
US (Dallas)
US West (Oregon)
US (Chicago)
US West (Northern California)
UK (but separate account required)
EU (Ireland)
Asia Pacific (Singapore)
Asia Pacific (Tokyo)
Asia Pacific (Sydney)
South America (Sao Paulo)
20

21. Addressing Security Issues
• Rackspace
•
•
•
•
Isolated networking
Windows Firewall
Dedicated Servers
Private Cloud (software)
• Amazon
•
•
•
•
Dedicated servers
Windows Firewall
Private Cloud
VPN Gateway
21

22. Other Considerations
• Image Backups
• Can be created at any time
• Minimal cost to store images (~$.10/GB/month)
• File Level Backups
• Backup tools
• Can set up separate drive and clone it
• Where are you writing your scripts?
• Do you need to spin up a cloud machine to create scripts or just for
execution?
• Managed service level and/or monitoring agreements for
additional cost
• Software to remind you if you leave instances running
(or auto shut them down)
22

23. Road Map to Set up your Own Performance
Test Lab in 24 Hours or Less
Sign up for
Account
Modify
Firewall, User
Access & install 3rd
party SW
Verify Connectively
to Controller and
Image LG
Some magic here
Launch Controller
Server
Install LG Software
Install Controller
Software (and
license)
Launch Load
Generator Server
Spawn N copies
of LG Image
Start Testing
23

24. Step 1: Start in the Right Place
24

25. Step 2: Know What you Need
25

26. Step 3: Pick Good Name and
Enough Space
Windows O/S needs 30GB space
26

27. Step 4: Wait
• Depending on size and number of instances will take between
5-15 minutes before the instances are available for use
• Best to create 1 instance, set it up and then clone it
• Setup includes
•
•
•
•
User access (default only Admin)
Set Admin Password
Configure Firewall
Install software
• In addition to testing software will also need browsers (only IE with
no add-ins), add-ins for browsers (java, flash, etc.)
• General use software such as adobe reader, zip file handler, excel,
etc.
27

28. Step 5: Use (and Stop using)
• You will be charged for usage as soon as the machine starts
spinning up
• You will be charged for any portion of an hour
• If you are not using a machine, remember to stop it
• 23 cents/hr. is nothing until you accidently leave 20 machines
running over the weekend
• Consider Saving results & doing analysis locally
• Can also have separate drive that you can un-mount from cloud
machine, turn off machine and access data from local machine
• Work on scripts locally or use smaller instance in cloud for
script development
• Consider having license server in cloud (on small instance that
is always on)
28