Performance and Load
Testing Process.
How to plan it
www.qaexperts.pro

22
12+ years general working experience in IT/QA (40+ projects)
----------
•QA Consultant / QA Manager at Ciklum Interactive
•Load Testing Manager (3+ years, 15+ projects)
•Automated Testing Manager
•Co-owner at www.qaexperts.pro
----------
Skype: vladimir.primakov
Linkedin: ua.linkedin.com/in/vladimirprimakov/
Email: v.v.primakov@gmail.com
Some Words About Me
Volodymyr Prymakov (Vladimir Primakov)
www.qaexperts.pro

33
• Introduction - 5 min
• Main Part - 30 min
• Questions - 5 min
Presentation’s Structure
Presentation Plan
www.qaexperts.pro

44
Assumptions
We assume that:
• Load & Performance testing tools are already chosen and used.
• Load testing framework and infrastructure is build and fully functional.
• Load testing team is already formed
www.qaexperts.pro

55
Load Testing Process
1. Planning
2. Evaluating Technical Risks
3. Prerequisites solving
4. Script implementation & adjustment
5. Test execution, analysis, and reporting
6. Summary reporting and conclusions
www.qaexperts.pro

66
1. PLANNING
www.qaexperts.pro

77
PRODUCT AND SCOPE
www.qaexperts.pro

88
Product Idea:
Business domain, main specialties
PRODUCT AND SCOPE
Solution Architecture.
The underlying technology.
Pointer to Weak areas.
www.qaexperts.pro

99
Product and product parts
IN and OUT of scope.
Priority
PRODUCT AND SCOPE
3D-Party services
IN and OUT of scope.
Mockups. Impact?
www.qaexperts.pro

1010
PRODUCT AND SCOPE
www.qaexperts.pro
60%
30%
2%8%
CLIENT PLATFORMS?

1111 www.qaexperts.pro
PRODUCT AND SCOPE
COMMUNICATION TECHNOLOGY
• Communication model and implementation for
a) client <-> server side
b) server-side <-> server-side
• Protocols/exchange formats used in the communication
• If there is any repetitive calls available in the communication?
HTTP(S)
(85%)
Web-sockets
(10%)
TCP/IP
(5%)
TLS
HTML
Strings
Binary
XML
JSON

1212
PRODUCT AND SCOPE
USAGE FLOWS / USER ACTIONS
www.qaexperts.pro

1313
PRODUCT AND SCOPE
USAGE FLOWS / USER ACTIONS
www.qaexperts.pro
•Types of flows
• Logged in and anonymous
• Main Operations
• Navigational
•Approach (Stability and Traceability):
•Try to isolate testable operation in a single flow
•Make flows as shorter as you can
•Avoid dependencies between flows

1414
PRODUCT AND SCOPE
USAGE FLOWS / USER ACTIONS
www.qaexperts.pro
Flow Type Flow %
Anonymous
Registration 3%
Main Pages Navigation 10%
Categories Navigation 15%
Search and Navigate Item 20%
Search and Buy item 15%
Search and Add to Cart / Remove from
cart 15%
Compare Items 5%
Logged in
Add comment to / Ask question about
an item 5%
Add item to my wish list 5,00%
Navigate to wish list 5,00%
Edit Profile 2%
Etc… …
SUM 100,00%
Registration Flow
1. Open main page
2. Go to the registration page
3. Fill in all the required data and
click Register
4. Get the corresponding
registration activation email -
follow the registration activation
link and login
Example:

1515
TEST ENVIRONMENTS
www.qaexperts.pro
Test environments for:
1. Evaluating the product / Technical Risk
2. Script implementation
3. Load test execution
Difference between environment for test
execution and target production environment?
The corresponding risks.
Possibility to minimize the difference.

1616
LOAD PROFILES
TYPES OF LOAD TESTS
www.qaexperts.pro
•10-100% Usual Load
•Stress Load
•Capacity Testing
•Spike Stress Load
•Soak Testing (24h-7d)
•Volume Testing
Load profiles Types of load tests

1717
EXPECTED LOAD
PASS-FAIL CRITERIA
www.qaexperts.pro
•Client-side criteria
•Server-side criteria
•Test Data

1818
EXPECTED LOAD
PASS-FAIL CRITERIA
www.qaexperts.pro
Client
Which is actually:
a) Number of registered users or
b) Number of users per day or
d) Correct number of concurrent users 
Response time for a operation or a request?
Number of concurrent users
=100000
Response time: 2 Sec

1919
EXPECTED LOAD
PASS-FAIL CRITERIA
www.qaexperts.pro
CLIENT-SIDE CRITERIA
•Max number of concurrent users (anonymous and logged-in, active
and passive)
•% of Mobile/web/desktop/terminal clients
•Max rate of main actions/operations per min, e.g.:
100 logins/min, 10 registration/min
•Max response times for key pages and operations, e.g.:
login – 3 sec, registration – 5 sec
•Acceptable % of fails, e.g.: 2%
Availability of such statistics from web-logs, Google analytics, and other tools/places?

2020
EXPECTED LOAD
PASS-FAIL CRITERIA
www.qaexperts.pro
SERVER-SIDE CRITERIA
•Max Resource Consumption:
CPU (e.g. 80%), RAM (e.g. 80%), Disk in/out, Network, etc.
These parameters should be monitored within every node of a certain
component
•Max Errors Rate
(in logs, etc.)
e.g. 0.5%

2121
EXPECTED LOAD
PASS-FAIL CRITERIA
www.qaexperts.pro
TEST DATA
•Users
•User’s Data
•Product’s Data
---------------------------
1. Type of Data,
2. Size and Amount
of Data

2222
OTHER NUANCES
www.qaexperts.pro
•Scheduling services?
•Client’s internet
connection speed?
•Static content hosting:
CDN or own servers?
•Target region (USA,
Europe, etc.)?

2323
ORGANIZATIONAL MOMENTS
www.qaexperts.pro
•Project Milestones?
•Date and Time frames for
load testing runs?
•Stakeholders?
•Organizational, Business,
Technical, and Server-side
support contacts

2424
• Difference between Target
and Test environments
• Risk of using service-
mockups
• Risk of negatively impacting
the production environment
or 3d party services
• Wrong target load
expectations
• …
www.qaexperts.pro
TYPICAL RISKS

2525
• Product & Solution
successfully passed UAT
• All prerequisites are solved
• No unexpected deploys
• …
www.qaexperts.pro
DEPENDENCIES / ASSUMPTIONS

2626
2. EVALUATING
TECHNICAL
RISKS
www.qaexperts.pro

2727
3.
PREREQUISITES
SOLVING
www.qaexperts.pro

2828 www.qaexperts.pro
PREREQUISITES SOLVING
PLANNING IS DONE!
Non-functional requirements
and user flows are defined.

2929 www.qaexperts.pro
PREREQUISITES SOLVING
PREPARE TEST ENVIRONMENT!
• Test environments availability
and access. Whitelisting load
injectors.
• Test environment and product
stability.
• Sever-side monitoring tools.
• Test environment alignment to
the production environment.
• Test users and test data.

3030
4.
SCRIPT
IMPLEMENTATION
www.qaexperts.pro

3131 www.qaexperts.pro
SCRIPT IMPLEMENTATION
1. Type of scripts:
• Logged-in/Anonymous
• Main flows scripts,
• Navigational Scripts,
• Repetitive calls scripts
2. Adjust scripts to the target
environments
3. Regularly Update scripts to
product changes
4. Define and follow script
readiness criteria

3232
5.
TEST EXECUTION,
ANALYSIS, AND
REPORTING
www.qaexperts.pro

3333
www.qaexperts.pro
PLAN TEST EXECUTION
a) Plan load gradually (to eliminate Risks):
10%
25%
50%
75%
100%
b) Or start load from Capacity test with long ramp-up
period (to get fast results)
Define load schedule: Types of load tests, dates, and
timeframes for test runs

3434 www.qaexperts.pro
• Type of test/Goal
• Date & Time of test
• Server sider conditions and configuration; product
version.
• % of load
• Ramp-up, Sustain, Ramp-down
• Scripts (Flows) to run
• Number of concurrent users in general and per script
• Expected operations rate
• Responsible specialists and their contacts in case of
emergency.
PLAN A RUN
PLAN TEST EXECUTION

3535 www.qaexperts.pro
TEST EXECUTION
• Make sure the server side support’s specialists are available
• Do smoke test
• Run Test:
• Monitor Test product health:
• Client
• Response time growth
• Error rate growth
• Error types
• Server
• Resource consumptions
• Errors
• Servers availability
• Communicate the results in real time
• Be ready to Stop Test in case of Emergency!
• Monitor Load Injectors health
• If possible, play manually with the product under load

3636 www.qaexperts.pro
TEST RESULTS ANALYSIS
AND REPORTING
Client Side:
• Response time-based capacity point
• Errors-based capacity point
• Failed scripts and transactions
• Slow scripts and slow transactions
• Errors details: types and amount
• Max transactions/operations rates reached.
• Timely trends and correlation with concurrent
users number
• Main bottlenecks concussions
Server-side:
• Resources consumptions
• Errors and log-messages
• Slow DB transactions
• Correlation with client side metrics

3737 www.qaexperts.pro
Execute
Load Test
Meet
Pass
Criteria?
Do
Profiling
Load Test
Fix
Solution/
Product
TEST EXECUTION PROCESS
Ready For
Release
YES
NO
If possible, the final step of load testing is testing on the prod environment under
conditions similar to the real ones.
Update
Scripts
Analyze
Results

3838
6.
SUMMARY
REPORTING AND
CONCLUSIONS
www.qaexperts.pro

3939
www.qaexperts.pro
SUMMARY REPORTING
AND CONCLUSIONS
1. Compare test runs: types of tests,
server-side conditions, client and
servers-side metrics
2. Conclusion about readiness to release.
3. Recommendations on further steps:
a. Next load tests required.
b. How to survive the expected load in
case of not meeting the PASS criteria.

4141
GENERAL SCHEDULE & ESTIMATION
Weeks 1 2 3 4 5 6 7 8 9
Planning
Evaluating Technical Risks
Prerequisites solving
Script implementation &
adjustment
Test execution, analysis, and
reporting
Summary reporting and
conclusions
80%
50%
EFFORTS AND DURATION
www.qaexperts.pro

4242
LOAD TESTING IN SDLS
• In ongoing development – Verifying and validating
component, queue of components, and integration
related performance & robustness.
• Before Release – Verifying and validating the whole
product performance & robustness before release.
• Maintenance – Verify and validating architectural,
configurational, capacity-related, db-related, and
integration-related changes.
WHEN TO PERFORM
www.qaexperts.pro

4343
Thank you! ;)
QUESTIONS?