5. 5
Skills | Knowledge | Collaboration
Why performance testing is important
✓ 50% of frustrated users will visit another
website to accomplish their activity and 22%
won't return.
✓ 49% of respondents expect web pages to
load in under 2 seconds.
✓ 30% expect a 1-second response.
✓ 18% expect a site to load immediately.*
* http://docplayer.net/29696161-Performance-matters-key-consumer-insights.html
6. 6
Skills | Knowledge | Collaboration
From user perception point of view
What is a fast web-side?
< 1 second very fast
< 2 seconds quite fast
2-4 seconds acceptable
5-15 seconds slow
>15 seconds too slow
* https://www.hobo-web.co.uk/your-website-design-should-load-in-4-seconds/
8. 8
Skills | Knowledge | Collaboration
How smartphone users react to slow web-sites
Curse at their
phone, 23%
Scream at
their phone,
11%
Throw their
phone, 4%
Behave more
or less
normally,
62%
9. 9
Skills | Knowledge | Collaboration
Cost of poor performance
* http://www.webperformancetoday.com/2010/06/15/everything-you-wanted-to-know-about-web-performance/
If your average sales per hour is
$5,000
1 minute of downtime is
costing over $80
then
10. 10
Skills | Knowledge | Collaboration
Cost of poor performance
A 1-second page load delay equals*:
* http://www.webperformancetoday.com/2010/06/15/everything-you-wanted-to-know-about-web-performance/
16 % decrease in customer
satisfaction
11 % fewer page
views
7 % loss in
conversions
11. 11
Skills | Knowledge | Collaboration
Cost of poor performance
For example:
✓ John Lewis’s website went down around 3.20 pm on Black Friday - analysts estimated it could
cost £ 2.8m pounds.
✓ Web giant like Amazon would lose as much as $120 000 per minute of downtime.
✓ Microsoft Bing found that a two-second slowdown caused a 4.3 percent reduction in
revenue per user.
✓ Website Shopzilla reduced page load times from 7 seconds to 2 seconds, resulting in a 7–12
percent increase in revenue and 50 percent reduction in hardware costs.
* http://www.webperformancetoday.com/2010/06/15/everything-you-wanted-to-know-about-web-performance/
12. 12
Skills | Knowledge | Collaboration
When performance testing is needed
✓ The solution is already struggling from performance
problems.
✓ The solution have to deal with big amount of concurrent
users presently or in future (~ 300-500 concurrent users and
higher).
✓ The solution have a large database or should transfer or
process big amount of data in real time (~ 10 and more
concurrent users).
✓ The solution has complex architecture and a lot of internal
and external integrations running concurrently.
14. 14
Skills | Knowledge | Collaboration
✓ Performance Testing - the process of testing to determine the
performance of a software product (ISTQB Foundation).
✓ Performance testing – is a non-functional software testing technique
which determines responsiveness, stability, reliability and resource
usage of system under a certain user load (Wikipedia).
✓ Web load testing is nothing more than exercising a website under a
variety of production-like conditions to determine how it’s going to work
and to identify (and hopefully resolve) problems before your customers
find them (WebLoadTestingForDummies).
Performance testing. Definition
15. 15
Skills | Knowledge | Collaboration
Determine responsiveness, stability, reliability and resource usage of system
under a certain user load
Demonstrate that the system meets performance criteria (KPIs)
Compare different system configurations and versions to evaluate
performance improvement/degradation.
Performance trends tracking during the time
Determine system behavior under different load
Evaluate the system capacity
Scalability. Determine ability of a system to handle a growing amount of workload
Prepare the application for planned load (e-commerce: Black Friday, Marketing
Campaign. Finance: quarter/annual reporting, etc.)
Find which components of the system perform poorly under certain workload
Goals of performance testing:
16. 16
Skills | Knowledge | Collaboration
Slow sub-systems / functions (poor response)
Low capacity point
Configuration problems (web-server,
load balancers, db etc)
Dead-lock while simultaneous load
Flawed queue logic
Incorrect synchronization of recourses
Database issues e.g. size, indexing, replication
Memory, space and connections leaks
Poor network configuration
CPU, Memory utilization
Functionality bugs (how system should behave under overload, others).
Performance bottlenecks:
17. 17
Skills | Knowledge | Collaboration
Functional & Performance testing comparison
# Functional testing Performance testing
1
To verify the accuracy of the system against
expected results
To verify the behavior of the system at various
load conditions
2 Manual or automated Automated only
3 Could be done without special tools
Special set of tools is used including analyzing
and monitoring ones
4 One user performing all operations Several users performing desired operations
5
Involvement required from Customer, Tester
and Developer
Involvement required from Customers, Tester,
Developer, DB admins, DevOps
6
Test environment capacity/size could differ
from Production
Requires close to Production Test
environment!!!
based on: http://www.softwaretestinghelp.com/introduction-to-performance-testing-loadrunner-training-tutorial-part-1/
24. 24
Skills | Knowledge | Collaboration
Types of Performance Testing (load profiles)
Stress/Capacity test
Max Designed Operation Capacity
Volume test
8-72 hours or longer
+ Component Test
+ Reliability /
Recovery Test
Server-side performance
26. 26
Skills | Knowledge | Collaboration
Client-side performance testing - testing of one separate page load from client/browser side
Client-side Performance
27. 27
Skills | Knowledge | Collaboration
More detailed process of page load
Client-side Performance
28. 28
Skills | Knowledge | Collaboration
First thing the user sees
Client-side Performance
29. 29
Skills | Knowledge | Collaboration
Visual Experience
Client-side Performance
1. First Paint 2. First Contentful Paint
3. First Meaningful Paint 4. Visually Complete
based on: https://www.slideshare.net/nicjansma/measuring-real-user-performance-in-the-browser
31. 31
Skills | Knowledge | Collaboration
1. Run client-side performance of the page when there is 0 load.
2. Set performance metrics baselines based on step 1
3. Run client-side performance when there is different load on the server –
during Capacity/Load/Spike etc. tests
4. Compare results with baselines, analyze and summarize possible issues
Flow we suggest:
Client-side Performance
33. 33
Skills | Knowledge | Collaboration
Load testing metrics (synthetic monitoring)
Application-side Server-side
- Response time
- Throughput (rps/tps/tpm)
- Concurrent users
- Error rate (response code)
- Number of transactions passed/failed
- Network traffic
- CPU
- Memory
- Network
- Disk
- DB connections
- Logs error, warnings
34. 34
Skills | Knowledge | Collaboration
Response time = Latency (travelling across a network) + Processing time (system processing
of request)
• Average response time
• Peak response time (max)
• Response time with 95% or other percentile
Response time
Application-side metrics
35. 35
Skills | Knowledge | Collaboration
Throughput - how many simultaneous
requests/transactions per second/minute application can
handle
Note:
TPS could correlate with response time if requests are
consequent. The longer response – the lower tps
TPS does not correlate directly with response time if
requests are parallel.
consequent > 1 request – 1 sec response – 1 tps
parallel > 10 requests – 1 sec response – 10 tps
TPS could be improved by improving response time or
by increasing concurrent users
Throughput (rps/tps/tpm)
Application-side metrics
36. 36
Skills | Knowledge | Collaboration
Network traffic, Response codes, Error rate
Application-side metrics
Network traffic – shows how much data is flowing back
and forth from your servers (Kbytes or Mbytes / sec).
We can compare this metric to the response-time metric
to see how the throughput affects transaction
performance.
Error rate – is the mathematical calculation that
produces a percentage of problem requests compared to
all requests.
It is no standard for tolerable error rate. Some projects
consider 1% error rate successful in case the system can
handle maximum load without crash. Others consider
any errors. Still, few errors is not uncommon, especially
for large load.
37. 37
Skills | Knowledge | Collaboration
• You can monitor server-side metrics directly on the server (Linux, Windows)
• You can automate this process creating some monitoring agent to track metrics
• You can use one of monitoring tools:
- CloudWatch (Amazon)
- AppDynamics
- DynaTrace
- NewRelic
- Graylog, etc.
Server-side monitoring tools
*In addition to the server metrics monitoring during the load test (synthetic monitoring), monitoring tools allow
Real User Monitoring (RUM).
RUM is a type of performance monitoring that captures and analyzes each transaction by real users of a website or
application. Unlike synthetic monitoring, RUM never rests. It collects data from each user using every browser
across each request.
39. 39
Skills | Knowledge | Collaboration
Strong load testing tools should be able
Traffic recorder
Have IDE (console or GUI) which allows:
a) Create Requests of required protocol (HTTP, HTTPs, WS, WSS, JDBC, TCP, AJAX, etc.)
b) Support Transactions – to track time for all static data loading / redirections
c) Create Load Scenarios with ability of parametrization
d) Build different Load Profiles with rump-up and shut-down
e) Have debugger
Load runner engine
Distributed testing
Load test data saving (distributed), including client and server-side metrics
Load test data monitoring in real time
40. 40
Skills | Knowledge | Collaboration
- JMeter
- Gatling
- Locust
- The Grinder
- Apachebench
- Artillery
- Tsung
- Vegeta
- Siege
- Boom
- Wrk
Open source load testing tools
According to Load Impact tools comparison research*:
JMeter, Gatling, Grinder, Tsung and Boom all offer good performance, accuracy and reliability
Artillery, Locust and Siege have various issues with performance, accuracy and/or reliability
Performance-wise, Wrk and Apachebench are in a class of their own
NOTE: None of the tools tested can simulate thousands of VUs on a single machine without significant
degradation in measurement accuracy
* http://blog.loadimpact.com/open-source-load-testing-tool-benchmarks
45. 45
Skills | Knowledge | Collaboration
1. Planning
Identify Performance Acceptance Criteria and KPIs
• Response time for different type of transactions -> a user concern
• Throughput (tps/rps) -> a business concern
• Resource utilization -> a system concern
• Concurrent users number
• Accepted Error rate
• Accepted deviation for response time, resource utilization
• System behavior when overloaded
46. 46
Skills | Knowledge | Collaboration
1. Planning
Performance requirement analysis
Bad sample of requirements:
1. Response time should be no more than 4 seconds
2. System should be able to deal with 10 thousands
concurrent users
It is not clear whether:
• Response time should be 4 seconds for all requests?
• What if all response time will be 3.99 seconds?
• What if most request will response with 2 seconds but several with 6 seconds?
• What users do and how often? Open Main page or more?
• What if more than 10 ths. users? System should scale, or new users will be rejected or response time
just increases?
47. 47
Skills | Knowledge | Collaboration
1. Planning
Performance requirement analysis
Good sample of requirements:
1. Set average response time for different types of transactions:
• Simple navigation requests - 1 second
• Logging - 2 seconds
• Search and buy - 4 seconds
2. Set deviation - no more than 15%
3. Set failure rate - no more than 1%
4. Set CPU, memory, network, other server-metrics thresholds
5. Set response times, deviation, server-side metrics for different number of users - for
1k, 5k, 10k
6. System should reject new users in case of overload, showing informing message
48. 48
Skills | Knowledge | Collaboration
1. Planning
Identify the Test Environment
• Identify the logical and physical production architecture for performance testing
• Compare the both test and production environments while identifying the testing environment
TEST environment MUST BE the same as PROD (or run tests on PROD)
• Get resolve the environment-related concerns if any – using stabs for 3-rd parties or others
• Analyze whether additional tools are required for performance testing, like monitoring tools. Install such
tools.
Identify scope of load testing (product parts, 3-rd party services in/out of scope)
Identify technical nuances
• Scheduling services?
• Ping calls
• Client’s internet connection speed?
• Static content hosting: CDN or own servers?
• Target region (USA, Europe, etc.)?
49. 49
Skills | Knowledge | Collaboration
1. Planning
Plan and Design Tests
• Identify key usage scenario and workload (Load, Capacity, Spike, etc.)
• Define test data
• Establish metrics to be collected
What transactions to include:
1. Critical transactions
Example: System login, session support
2. Mostly used transactions by real users
Example: System login/logout, main page
3. Business required
Example: some specific feature
4. Risky transactions
Example: Checkout, payment
5. Heavy transactions
Example: File download/upload
50. 50
Skills | Knowledge | Collaboration
User Community Modeling Language (UCML) for Performance Test Workloads
1. Planning
51. 51
Skills | Knowledge | Collaboration
Prerequisites solving
1. Planning
• Add your load generator machine IP(s) into whitelisting (if needed)
• Request test users or create them by yourself
• Request test data (products names, files, etc.)
• Request 3-rd party dependent data (payment cards, etc.)
Configure load-generation environment
The output of this stage is prepared Performance Test Plan
• Environment capacity (CPU, memory) should be enough to run required number of users during
some time
• Setup Master-Slave architecture in case of distributed load testing
• Setup monitoring tools to track load server(s) health during tests running
52. 52
Skills | Knowledge | Collaboration
2. Implementing
• Develop performance scripts
• Put assertions points and wait timers to make it a real time scenario
• Run several smoke runs to calibrate scripts to the target environments in accordance with test design
Simulate real users behavior basing on Production usage statistic
53. 53
Skills | Knowledge | Collaboration
3. Executing
It very depends on a project, but still we can suggest next
How to choose and run correct load profiles
54. 54
Skills | Knowledge | Collaboration
0. Warm up your product servers before load tests run
- Run some short smoke load test for several users
3. Executing
55. 55
Skills | Knowledge | Collaboration
1. Run several baseline Load tests to determine benchmark performance metrics
levels:
- virtual users: 5-100
- duration: 30-60m
- rump-up – 10m; run – 20m; shutdown – 5m
!!! Run each test at least 2 but better 3 times to be sure in results
3. Executing
56. 56
Skills | Knowledge | Collaboration
2. Run the Capacity test profile and determine when Capacity point (system limit)
happens:
- virtual users: a lot, depends on the system
- duration: 2-3 hours
- rump-up – 2-3 hour; run – NA; shutdown – NA
!!! Run each test at least 2 but better 3 times to be sure in results
3. Executing
57. 57
Skills | Knowledge | Collaboration
How to understand that you reach Capacity point:
When transactions Response time increases sharply with load increase.
Note:
- although load increase, response time should not increase sharply. However, small
deviation is acceptable;
- response time also could start decrease dramatically, what means that you start receive
responses with errors.
When the Error rate increases with load increase
Note: often, small error rate is acceptable (1-3%), especially during large load tests.
Crash of the servers or one of them (web, application, DB, etc).
3. Executing
58. 58
Skills | Knowledge | Collaboration
Capacity point example
V users: 10 ths.
Status: failed
Error rate: test stopped on 28%
59. 59
Skills | Knowledge | Collaboration
Capacity point example
V users: 9 ths.
Status: failed
Error rate: test stopped on 49%
60. 60
Skills | Knowledge | Collaboration
Capacity point example
V users: 8 ths.
Status: passed
Error rate: 0.21% (500 errors
from 236 ths. transactions
61. 61
Skills | Knowledge | Collaboration
3. Run the Load test profile with:
- virtual users: 50-80% of Capacity point or/and required load number
- duration: 2-3 hours
- rump-up – 60m; run – 60m; shutdown – 20m
!!! Run each test at least 2 but better 3 times to be sure in results
3. Executing
62. 62
Skills | Knowledge | Collaboration
4. Run Component tests for each service / micro-service / function separately:
- could be run before or after Endurance test, especially if issues for some service(s) were
found
- virtual users: firstly, like for Capacity test, then like for Load profile
- duration: 1-2 hours
- rump-up, run and shutdown timings depend on component test profile
!!! Run each test at least 2 but better 3 times to be sure in results
3. Executing
63. 63
Skills | Knowledge | Collaboration
5. Run the Endurance test profile:
- virtual users: 20-40% of Capacity point
- duration: 8-72 hours
- rump-up – 30m; run – 23р; shutdown – 30m
!!! Run each test at least 2 but better 3 times to be sure in results
3. Executing
64. 64
Skills | Knowledge | Collaboration
6. Run other types of load testing and play with your scenarios combinations.
- volume test, Spike, Reliability/Recovery, Volume tests
- each of these tests could be run on the first stages, if it is prioritized be business
- run load tests for different user flows combinations
!!! Run each test at least 2 but better 3 times to be sure in results
3. Executing
65. 65
Skills | Knowledge | Collaboration
• In ongoing development -> Verifying and validating
component, queue of components, and integration
related performance & robustness
• Before release -> Verifying and validating the whole
product performance & robustness before release.
• Maintenance -> Verifying and validating architectural,
configurational, capacity-related, db-related, and
integration-related changes
When to perform
3. Executing
66. 66
Skills | Knowledge | Collaboration
• Collect and analyze load tests data
• Investigate possible bottlenecks (memory, disk, processor,
process, cache, network, etc.) resource usage like (memory,
CPU, network, etc.)
• Generate the Performance analysis reports
Note: The report form is
- a performance test summary report
- transactions details, hardware utilization, etc.
- the comparison of actual and expected KPIs
• Based on the analysis prepare recommendation report
• Share report with the team and stakeholders
4. Analyzing and reporting
68. 68
Skills | Knowledge | Collaboration
5. Continuous Integration
or
Development
Continuous Integration
Build. Unit test
Deploy to Dev Server
Integration Test
Deploy to Test/QA
Server
Automated Functional
Tests
Deploy to Load Server
Performance test in CI +
monitoring
Manual QA Test
Promote to
Stage / Pre-Prod
Ideally, if you have separate
environment for load testing
Note: you can implement
load tests on Stage too, if
it is the same as Prod
Highlight a value, combination of benefits and approaches
- Endurance test – тестирование стабильносты. Профиль тот же что при лоад, но время дольше.
При пиках интересно сможет ли система восставновится после пика, освободить ресурсы и т.д. Пики можно делать только для какого-то сценария, либо для всех.
Стресс тестирование – найти точку насыщения, когда нагрузка достигла критической, время отклика начинает расти хоть до этого не увеличивалась.
Не ограничивайтесь только этими профилями, стройте зависимо от вашей системы
- TPS could be improved by improving response time or by increasing concurrent users (кассира заставить работать быстрее, либо увеличить количество рабочих кас)
- TPS could be improved by improving response time or by increasing concurrent users (кассира заставить работать быстрее, либо увеличить количество рабочих кас)