Virtualization, Cloud Deployments, and Cloud-Based Tools have challenged and changed performance testing practices. Today’s performance tester can summons tens of thousands of virtual users from the cloud in a few minutes at a cost far lower than the expensive on-premise installations of yesteryear. Meanwhile, systems under test have changed more. Updated software stacks have increased the complexity of scripting and performance measurement, but the biggest changes are in the nature and quantities of resources powering the systems. Interpreting resource usage when resources are shared on a private virtualization platform is exceedingly difficult. Understanding resources when they live in a large public cloud is impossible.

1. Performance Testing in
New Contexts

2. • @ericproegler
• Performance Engineer
• 13 years in (back-end) performance,
20 in software
• Speak Easy Mentor (speakeas.ie)
• AST Board (associationforsoftwaretesting.org)
• WOPR Organizer (performance-workshop.org)

5. How I Learned to Perf Test…
• Monitor everything: discrete and limited resources
(Windows 2000, on Pentium 3)
• Injectors every 50-100 virtual users
• Mercury, Silk, QALoad, Rational Tools

6. How Is It Holding Up?
• Still lots of projects where we can monitor discrete
and limited resources, to some extent
• Injectors every 500-1000 virtual users
• More on tools later, but there have been a lot of
changes…

9. Resources, Shared
• Magical SAN thinking
• Virtualization at first:
We can watch the Virtualization Host
Resources and model?
• Production and the problem of shared
resources

12. Resources, Bundled
• Resources Are Hard to Trace
• Dynamic reallocation: How vSphere works
• Maybe we have to trust the Admin? Or look
over their shoulder? Or enlist them?

13. Resources, Evaluated
• CPU Ready requires host-level knowledge
• The paradox of fewer vCPUs
• Overcommitted resources – the patching
story

16. Resources, Abstracted
• Resources Are Impossible to Trace
• Now we HAVE to trust the Admin
• Even with our injectors…

19. Resources, Outsourced
• Now there isn’t an admin we can talk to, and
it’s not possible to see
hardware/virtualization host layer
• Lots of claims about scalability/elasticity,
though most systems still engineered with
specific numbers of servers, config files

20. It’s All Out of Our Hands
•We don’t know who the admin is
•We have response time against load models –
but no resources. Watch carefully for these to
increase.

21. Story time!

23. Large Scale Mobile App
• Linked to prime-time TV -- Asks trivia questions during
live TV shows; gamers score points they can use to buy
stuff; TV show sponsors pay for “sticky viewers”
• Complaints by users not being able to connect and answer
questions; seems to occur above 20k concurrent users, but
trigger conditions not evident

24. Large Scale Mobile App
• Growing fast, need to be able to scale to support mega shows
• Applications run on Amazon EC2
• Startup budget; can’t afford commercial tools. Found a little to
use a traditional tool just for monitoring
• Want to start with 20k users, eventually run with 50k users

26. Load Testing in the Cloud
• BlazeMeter is Jmeter in the cloud.
• Jmeter/Selenium/Gatling/Grinder/etc is free –
like a puppy!

28. Response Times (milliseconds)
Request/Action #Samples Average Median 90% Line Min Max Hit / s KB / s
ALL Requests 1,734,829 24.6 24.5 61.7 2 1038 945 11,949
Get_Settings 20,000 13.6 13.6 16.5 10 45 20 70
Get_event_data 20,000 9.0 8.9 12.9 7 60 20 24
Information_about_t
he_current_game
20,000 23.6 23.5 29.4 15 78 20 18
LANDING_HTML_PA
GE
20,000 11.3 11.1 117.7 5 152 20 36
Query Game Phase 1,112,998 28.1 28.1 48.1 18 667 618 569
Vote 181,831 35.6 35.7 61.3 13 245 151 73

32. Card Time Votes
1 14:13:35.436 1494
2 14:16:35.665 5327
3 14:19:35.892 9048
4 14:22:36.142 12594
5 14:25:36.423 15997
6 14:28:36.707 18086
7 14:31:27.027 18085
8 14:34:37.366 18080
9 14:37:37.640 18082
10 14:40:37.961 18080
11 14:43:38.256 1
Total 134,874
Actual
number of
votes in 30
minutes
Expected
number of
votes

33. My Takeaways
• Used to the level of information traditional tools
provide; had to figure out how to get it (run
transform on several hundred MB of xml, thin data
to get to Excel-sized chunks, etc).
• Would have started with a database if we could do it
again, Tableau for visualization
• A good tool to add to the toolbox for web sites at
larger scales

34. Story time!

35. E-Commerce Site
• Wants to test with 5,000 virtual users
• Number of VUs a little larger than we were
confident we had computation and bandwidth to
support (at the planning stages)
• Customer not interested in standing up load
generators
• Some budget for tools
• Ended up being a config (CDN) and ops (DR site)
test project

36. Injection in the Cloud, First Hand
• NeoLoad – a tool we already knew:
– Affordable compared to other commercial tools
– Great analysis engine
– Credits system, rental terms
– Medium and Large Injectors, US/EU/Asia/etc
• Called 5,000 Users From Geographically
Distributed Engines
– Incoming Bandwidth can be an issue…

37. My Takeaways
• Was actually pretty seamless, analysis was as
easy as ever
• Buy plenty of credits, so you don’t have to
ration them

40. What I’m Doing Lately
• Always Cloud Injectors
• Geographic Diversity is
table stakes, and
necessary for complex
sites (Users, CDN…)

42. What’s Next?
• Automation Squared -> Automation Cubed
• An API Manifest converted to tests
• Micro-Perf Tests, every build, or constantly
• What do you think the future looks like?

43. Thanks!
@ericproegler
associationforsoftwaretesting.org
performance-workshop.org
.com
See you next time!