EuroSTAR Software Testing Conference 2008 presentation on Performance Testing in 5 Steps - A Guideline to a Successful Load Test by Mieke Gevers. See more at conferences.eurostarsoftwaretesting.com/past-presentations/

1. Performance Testing in 5 Steps:
'A Guideline to a Successful Load
Test'
Mieke Gevers

2. Mieke Gevers, AQIS - Belgium

3. © 2008-2009 AQIS bvba 3
Agenda
1. What is load testing?
2. Methodology
3. Conclusion

4. © 2008-2009 AQIS bvba 4
Agenda
1. What is load testing?
a. Terminology
b. Challenges
2. Methodology
3. Conclusion

5. © 2008-2009 AQIS bvba 5
1. Terminology
• "load testing" usually is defined as the
process of exercising the system under
test by feeding it the largest tasks it can
operate with.
• Load testing is sometimes called volume
testing, or longevity/endurance testing.

6. © 2008-2009 AQIS bvba 6
1. Terminology
• “Stress testing” is testing by trying to break
the system under test by using
overwhelming resources or by taking
resources away from it (in which case it is
sometimes called negative testing).

7. © 2008-2009 AQIS bvba 7
• Failover Tests : verify of redundancy
mechanisms while under load.
• Can the system-under-test recover &/or react
quickly enough? Can the other web servers
(load balancers) handle the sudden dumping of
extra load?
• Advantage: Allowing technicians to address
problems in advance, in the comfort of a testing
situation, rather than in the heat of a production
outage.
1. Terminology

8. © 2008-2009 AQIS bvba 8
• Soak Tests ~ Endurance Testing : running
a system at high levels of load for
prolonged periods of time.
• Ideal for finding memory leaks and other
defects which make a system prone to
crashing/breaking after a certain number
of iterations
1. Terminology

9. © 2008-2009 AQIS bvba 9
1. Terminology
• Network Sensitivity Tests : are tests that
set up scenarios of varying types of
network activity (traffic, error rates...), and
then measure the impact of that traffic on
various applications that are bandwidth
dependant.
• eg. Chat applications, Streaming Media…

10. © 2008-2009 AQIS bvba 10
1. Terminology
• Targeted Infrastructure Test : are Isolated tests
of each layer and or component in an end to end
application configuration. It includes all
components of the AUT infrastructure and
beyond.
• eg. includes communications infrastructure,
Load Balancers, Web Servers, Application
Servers, Crypto cards, Citrix Servers,
Database…
• Goal: identifying any performance issues that
would fundamentally limit the overall ability of a
system to deliver at a given performance level.

11. © 2008-2009 AQIS bvba 11
1. Terminology
Time
Predicted
point of failure
Upgraded Capacity
Existing Capacity
Lead
TimeTrigger
Point
Anticipated
Peak Loads
Work
load
Usable
Reserve
Capacity
Point Response Time
degradation becomes
noticeable
Current
load
©copyright 2004-2005 by Wilson Mar. All rights reserved.
Performance testing is the overall process,
Stress Test
Load Test
Scalability

12. © 2008-2009 AQIS bvba 12
2. Challenges

13. © 2008-2009 AQIS bvba 13
2. Challenge 1 : Communication
The requirements
specification was
defined like this
The developers
understood it
in that way
This is how the
problem was
solved before.
This is how the
problem is
solved now
That is the program
after debugging
This is how the program
is described by
marketing department
This, in fact, is what the
customer wanted … ;-)

14. © 2008-2009 AQIS bvba 14
2. Challenge 2 : HW & SW

15. © 2008-2009 AQIS bvba 15
2. Challenge 2 : HW & SW
Security

16. © 2008-2009 AQIS bvba 16
2. Challenge 3 : User Nature

17. © 2008-2009 AQIS bvba 17
2. Challenge 3 : User Nature

18. © 2008-2009 AQIS bvba 18
2. Challenge 3 : User Nature

19. © 2008-2009 AQIS bvba 19
Response Times: The Three Important Limits [Miller 1968; Card et al. 1991]:
•0.1 second
limit for having the user feel that the system is reacting instantaneously
•1.0 second
limit for the user's flow of thought to stay uninterrupted, even though the
user will notice the delay.
•10 seconds
limit for keeping the user's attention focused on the dialogue. For longer
delays, users will want to perform other tasks while waiting for the
computer to finish, so they should be given feedback indicating when the
computer expects to be done.
2. Challenge 3 : User Nature

20. © 2008-2009 AQIS bvba 20
• Computer scientists identify future IT
challenges
Goals for IT using the power of quantum
physics, building systems that can't go wrong
“Ambitious goals include harnessing the power
of quantum physics, building systems that
can't go wrong, and simulating living creatures
in every detail.” By Peter Sayer, IDG News Service January 25, 2005
2. Challenge 4 : Future

21. © 2008-2009 AQIS bvba 21
2. Challenge 5 : Resources
• HW
• Infrastructure
• Access
• People
• Time
• ....

22. © 2008-2009 AQIS bvba 22
Agenda
1. What is load testing?
2. Methodology
3. Conclusion

23. © 2008-2009 AQIS bvba 23
3. Methodology
Analyze

24. © 2008-2009 AQIS bvba 24
Documentation

25. © 2008-2009 AQIS bvba 25
1. Determine system performance, response time, and
throughput under a specific load
2. Testing the ability to handle load (and stress) to identify
bottlenecks
3. Evaluating whether system resources are being
utilised efficiently
4. Testing system robustness and capability to recover
from errors
5. Testing across different configurations or versions
6. Testing systems for scalability
1. Goals

26. © 2008-2009 AQIS bvba 26
Examine SUT
• Architecture
Overviews
• Deployment Topology
• 3rd party components
and SLA’s
• Firewall capacity
• Load balancing
• Connectivity
• Network
• Session management
• All queues
• Caching models
• Security methods
• Fail over mechanisms
• Redundancy
• Bandwidth
• ……

27. © 2008-2009 AQIS bvba 28
5 steps
Analyze
Plan/Define/Design scenario's

28. © 2008-2009 AQIS bvba 29
Design and Model
1. Transactions and Workflows scenario's
2. User profiling and modeling
3. Workload scenario's
4. Test environment
5. Test data

29. © 2008-2009 AQIS bvba 30
1. Transition matrix
online shopping visit
 Make Flow-charts of Functional Paths through the
application-under-test.
 Break this down in Logical modules = transactions

30. © 2008-2009 AQIS bvba 31
2. User Profiling
• User activities, transactions and usage
patterns
• Client platforms and preferences
• Client Internet access speeds and browser
types
• User geographic locations

31. © 2008-2009 AQIS bvba 32
3. Workflow Modeling
• Size of customer base
• Growth factor
• Site arrival rates and site abandonment
• Think times and latency
• Background noise

32. © 2008-2009 AQIS bvba 33
4. Test environment
• Production replica
–Expensive, usually not possible
• Scaled down
–Extrapolation factor
• Actual production equipment
–For new applications

33. © 2008-2009 AQIS bvba 34
5. Test data
• Data
• Randomized
• Database
• Cookies, Session ID’s, Hidden ID’s, values
• Certificates
• Security settings
• IP addresses identifications
• User specifics identifications, PKI’s

34. © 2008-2009 AQIS bvba 35
6 steps
Analyze
Plan/Define/Design scenario's
Build/Record scenario’s

35. © 2008-2009 AQIS bvba 36
Load Test scenario's
• Program Perl, Java
• Stubs
• HW devices
• FreeWare tools
• Commercial Tools (capture & playback)
•...

36. © 2008-2009 AQIS bvba 37
How to reproduce correct web
behavior?
• Correct environment simulation
– Correct protocol (Business level)
– Speed
– UI (Browser/MMS/..)
– Security simulation/usage
– Calculating the amount of virtual users and
the arrival rate
– ...

37. © 2008-2009 AQIS bvba 38
5 steps
Analyze
Plan/Define/Design scenario's
Build/Record scenario’s
Baseline + Load Test

38. © 2008-2009 AQIS bvba 39
What is a baseline?
• Baseline with one user
• Monitor the involved back-end
systems
• Can the load test achieve the goals
• Test-runs

39. © 2008-2009 AQIS bvba 40
Load Testing: How?

40. © 2008-2009 AQIS
bvba
41
Users Internet Firewall
Load
balancers
Web
Servers
Application
Servers
DB ServersLAN

41. © 2008-2009 AQIS bvba 42
Monitoring

42. © 2008-2009 AQIS bvba 43
What to Measure
 Maximum login requests (per min/per sec)
 Average session length (at peak)
 max. Concurrent sessions
 Average pages per session
 Average hits per page
 Average request distribution
 Average size of a page & specific size of a page
 Arrival rates
Tip: Session length(in min)= Max_concurrent_sessions/
Login_requests_per_min
Metrics

43. © 2008-2009 AQIS bvba 44
5 steps
Analyze
Plan/Define/Design scenario's
Build/Record scenario’s
Reporting
Baseline + Load Test

44. © 2008-2009 AQIS bvba 45
Analysing Results
– Correlation of monitor and load test data –
root cause analysis

45. © 2008-2009 AQIS bvba 46
6 steps?
Analyze
Plan/Define/Design scenario's
Build/Record scenario’s
Baseline + Load Test
Reporting
Iteration-Monitor

46. © 2008-2009 AQIS bvba 47
Tips
• Be aware of constant changes
• Constantly training
• User behaviours
• Increase of transactions (seasonal –
permanent)
• Use Common Sence

47. © 2008-2009 AQIS bvba 48
 Capacity planning for “Web performance”, Menascé
 Building High-Scalability Server Farms, 1999 Microsoft Corporation
 Detecting System Bottlenecks in Sites Using Site Server 3.0 Commerce
Edition, 1999 Microsoft Corporation
 Design for scalability, IBM High-Volume Web site team, December 1999
http://www7b.boulder.ibm.com/wsdd/library/techarticles/hvws/scalability.html
 Performane Testing Guidance for Web applications, Scott Barber
 ISO 9126, ISO/IEC 12207:2008, International Organization for
Standardization
Resources

48. © 2008-2009 AQIS
bvba
49
Questions &
Thank you !
Mieke Gevers
info@aqis.eu
www.aqis.eu
Agile Quality in
Information Systems