Dakiry, profile picture
Uploaded byDakiry
962 views

Микола Ковш “Performance Testing Implementation From Scratch. Why? When and How?”

This document discusses the importance of performance testing and provides an introduction to the topic. It notes that performance testing determines how a system behaves under different loads and helps identify bottlenecks. The document outlines why performance testing is important from a user experience perspective, discussing metrics like page load times and the financial costs of poor performance. It then covers various performance testing approaches, targets, levels, and common metrics used to evaluate performance.

Embed presentation

1 Skills | Knowledge | CollaborationSkills | Knowledge | Collaboration PerformanceTestingImplementationFromScratch.Why? WhenandHow? by Mykola Kovsh
2 Skills | Knowledge | Collaboration Speaker info Hi, I’m Mykola Kovsh PerformanceTestEngineer PerformanceTesting Departmentat Testing Centerof Excellence, Ciklum, Ukraine •4 yearsin IT •Performance/ Automation/ Manualtesting www.linkedin.com/in/mykolakovsh/
3 Skills | Knowledge | Collaboration Content Performance testing engineer skills5 Why performance testing is important?1 Introduction to performance testing. Load profiles, metrics2 Implementation process4 Performance testing tools3 Links and Q&A session
4 Skills | Knowledge | Collaboration 1. Why performance testing is important?
5 Skills | Knowledge | Collaboration Why performance testing is important ✓ 50% of frustrated users will visit another website to accomplish their activity and 22% won't return. ✓ 49% of respondents expect web pages to load in under 2 seconds. ✓ 30% expect a 1-second response. ✓ 18% expect a site to load immediately.* * http://docplayer.net/29696161-Performance-matters-key-consumer-insights.html
6 Skills | Knowledge | Collaboration From user perception point of view What is a fast web-side? < 1 second very fast < 2 seconds quite fast 2-4 seconds acceptable 5-15 seconds slow >15 seconds too slow * https://www.hobo-web.co.uk/your-website-design-should-load-in-4-seconds/
7 Skills | Knowledge | Collaboration
8 Skills | Knowledge | Collaboration How smartphone users react to slow web-sites Curse at their phone, 23% Scream at their phone, 11% Throw their phone, 4% Behave more or less normally, 62%
9 Skills | Knowledge | Collaboration Cost of poor performance * http://www.webperformancetoday.com/2010/06/15/everything-you-wanted-to-know-about-web-performance/  If your average sales per hour is $5,000 1 minute of downtime is costing over $80 then
10 Skills | Knowledge | Collaboration Cost of poor performance  A 1-second page load delay equals*: * http://www.webperformancetoday.com/2010/06/15/everything-you-wanted-to-know-about-web-performance/ 16 % decrease in customer satisfaction 11 % fewer page views 7 % loss in conversions
11 Skills | Knowledge | Collaboration Cost of poor performance For example: ✓ John Lewis’s website went down around 3.20 pm on Black Friday - analysts estimated it could cost £ 2.8m pounds. ✓ Web giant like Amazon would lose as much as $120 000 per minute of downtime. ✓ Microsoft Bing found that a two-second slowdown caused a 4.3 percent reduction in revenue per user. ✓ Website Shopzilla reduced page load times from 7 seconds to 2 seconds, resulting in a 7–12 percent increase in revenue and 50 percent reduction in hardware costs. * http://www.webperformancetoday.com/2010/06/15/everything-you-wanted-to-know-about-web-performance/
12 Skills | Knowledge | Collaboration When performance testing is needed ✓ The solution is already struggling from performance problems. ✓ The solution have to deal with big amount of concurrent users presently or in future (~ 300-500 concurrent users and higher). ✓ The solution have a large database or should transfer or process big amount of data in real time (~ 10 and more concurrent users). ✓ The solution has complex architecture and a lot of internal and external integrations running concurrently.
13 Skills | Knowledge | Collaboration 2. Introduction to Performance Testing
14 Skills | Knowledge | Collaboration ✓ Performance Testing - the process of testing to determine the performance of a software product (ISTQB Foundation). ✓ Performance testing – is a non-functional software testing technique which determines responsiveness, stability, reliability and resource usage of system under a certain user load (Wikipedia). ✓ Web load testing is nothing more than exercising a website under a variety of production-like conditions to determine how it’s going to work and to identify (and hopefully resolve) problems before your customers find them (WebLoadTestingForDummies). Performance testing. Definition
15 Skills | Knowledge | Collaboration Determine responsiveness, stability, reliability and resource usage of system under a certain user load Demonstrate that the system meets performance criteria (KPIs) Compare different system configurations and versions to evaluate performance improvement/degradation. Performance trends tracking during the time Determine system behavior under different load Evaluate the system capacity Scalability. Determine ability of a system to handle a growing amount of workload Prepare the application for planned load (e-commerce: Black Friday, Marketing Campaign. Finance: quarter/annual reporting, etc.) Find which components of the system perform poorly under certain workload Goals of performance testing:
16 Skills | Knowledge | Collaboration Slow sub-systems / functions (poor response) Low capacity point Configuration problems (web-server, load balancers, db etc) Dead-lock while simultaneous load Flawed queue logic Incorrect synchronization of recourses Database issues e.g. size, indexing, replication Memory, space and connections leaks Poor network configuration CPU, Memory utilization Functionality bugs (how system should behave under overload, others). Performance bottlenecks:
17 Skills | Knowledge | Collaboration Functional & Performance testing comparison # Functional testing Performance testing 1 To verify the accuracy of the system against expected results To verify the behavior of the system at various load conditions 2 Manual or automated Automated only 3 Could be done without special tools Special set of tools is used including analyzing and monitoring ones 4 One user performing all operations Several users performing desired operations 5 Involvement required from Customer, Tester and Developer Involvement required from Customers, Tester, Developer, DB admins, DevOps 6 Test environment capacity/size could differ from Production Requires close to Production Test environment!!! based on: http://www.softwaretestinghelp.com/introduction-to-performance-testing-loadrunner-training-tutorial-part-1/
18 Skills | Knowledge | Collaboration Performance testing Approaches Protocol-level Load Generation (HTTP, Web-sockets, TCP, JDBC, etc.) Device level Load Generation (User Interface)
19 Skills | Knowledge | Collaboration Performance testing Approaches Protocol-level Load Generation (HTTP, Web-sockets, TCP, JDBC, etc.) Device level Load Generation (User Interface)
20 Skills | Knowledge | Collaboration Performance testing Targets Server-side performance (web, app, database) Client-side (manual) performance Network performance Device performance (hardware: battery consumption, CPU, memory, etc.)
21 Skills | Knowledge | Collaboration Performance testing Targets Server-side performance (web, app, database) Client-side (manual) performance Network performance Device performance (hardware: battery consumption, CPU, memory, etc.)
22 Skills | Knowledge | Collaboration Server-side performance
23 Skills | Knowledge | Collaboration Levels of server-side performance testing • End-to-End load testing • Integration Load testing • Component/API Load testing • DB performance testing
24 Skills | Knowledge | Collaboration Types of Performance Testing (load profiles) Stress/Capacity test Max Designed Operation Capacity Volume test 8-72 hours or longer + Component Test + Reliability / Recovery Test Server-side performance
25 Skills | Knowledge | Collaboration Client-side (manual) Performance
26 Skills | Knowledge | Collaboration Client-side performance testing - testing of one separate page load from client/browser side Client-side Performance
27 Skills | Knowledge | Collaboration More detailed process of page load Client-side Performance
28 Skills | Knowledge | Collaboration First thing the user sees Client-side Performance
29 Skills | Knowledge | Collaboration Visual Experience Client-side Performance 1. First Paint 2. First Contentful Paint 3. First Meaningful Paint 4. Visually Complete based on: https://www.slideshare.net/nicjansma/measuring-real-user-performance-in-the-browser
30 Skills | Knowledge | Collaboration Client-side Performance Chrome performance testing
31 Skills | Knowledge | Collaboration 1. Run client-side performance of the page when there is 0 load. 2. Set performance metrics baselines based on step 1 3. Run client-side performance when there is different load on the server – during Capacity/Load/Spike etc. tests 4. Compare results with baselines, analyze and summarize possible issues Flow we suggest: Client-side Performance
32 Skills | Knowledge | Collaboration Load testing metrics
33 Skills | Knowledge | Collaboration Load testing metrics (synthetic monitoring) Application-side Server-side - Response time - Throughput (rps/tps/tpm) - Concurrent users - Error rate (response code) - Number of transactions passed/failed - Network traffic - CPU - Memory - Network - Disk - DB connections - Logs error, warnings
34 Skills | Knowledge | Collaboration Response time = Latency (travelling across a network) + Processing time (system processing of request) • Average response time • Peak response time (max) • Response time with 95% or other percentile Response time Application-side metrics
35 Skills | Knowledge | Collaboration Throughput - how many simultaneous requests/transactions per second/minute application can handle Note:  TPS could correlate with response time if requests are consequent. The longer response – the lower tps  TPS does not correlate directly with response time if requests are parallel. consequent > 1 request – 1 sec response – 1 tps parallel > 10 requests – 1 sec response – 10 tps  TPS could be improved by improving response time or by increasing concurrent users Throughput (rps/tps/tpm) Application-side metrics
36 Skills | Knowledge | Collaboration Network traffic, Response codes, Error rate Application-side metrics Network traffic – shows how much data is flowing back and forth from your servers (Kbytes or Mbytes / sec). We can compare this metric to the response-time metric to see how the throughput affects transaction performance. Error rate – is the mathematical calculation that produces a percentage of problem requests compared to all requests. It is no standard for tolerable error rate. Some projects consider 1% error rate successful in case the system can handle maximum load without crash. Others consider any errors. Still, few errors is not uncommon, especially for large load.
37 Skills | Knowledge | Collaboration • You can monitor server-side metrics directly on the server (Linux, Windows) • You can automate this process creating some monitoring agent to track metrics • You can use one of monitoring tools: - CloudWatch (Amazon) - AppDynamics - DynaTrace - NewRelic - Graylog, etc. Server-side monitoring tools *In addition to the server metrics monitoring during the load test (synthetic monitoring), monitoring tools allow Real User Monitoring (RUM). RUM is a type of performance monitoring that captures and analyzes each transaction by real users of a website or application. Unlike synthetic monitoring, RUM never rests. It collects data from each user using every browser across each request.
38 Skills | Knowledge | Collaboration 3. Performance testing tools
39 Skills | Knowledge | Collaboration Strong load testing tools should be able Traffic recorder Have IDE (console or GUI) which allows: a) Create Requests of required protocol (HTTP, HTTPs, WS, WSS, JDBC, TCP, AJAX, etc.) b) Support Transactions – to track time for all static data loading / redirections c) Create Load Scenarios with ability of parametrization d) Build different Load Profiles with rump-up and shut-down e) Have debugger Load runner engine Distributed testing Load test data saving (distributed), including client and server-side metrics Load test data monitoring in real time
40 Skills | Knowledge | Collaboration - JMeter - Gatling - Locust - The Grinder - Apachebench - Artillery - Tsung - Vegeta - Siege - Boom - Wrk Open source load testing tools According to Load Impact tools comparison research*:  JMeter, Gatling, Grinder, Tsung and Boom all offer good performance, accuracy and reliability  Artillery, Locust and Siege have various issues with performance, accuracy and/or reliability  Performance-wise, Wrk and Apachebench are in a class of their own  NOTE: None of the tools tested can simulate thousands of VUs on a single machine without significant degradation in measurement accuracy * http://blog.loadimpact.com/open-source-load-testing-tool-benchmarks
41 Skills | Knowledge | Collaboration Cloud-based tools: - BlazeMeter - LoadImpact - Loadstorm - Loader.io - Loadfocus, etc Commercial load tools Commercial Combiners: - HP Loadrunner - Microsoft TFS - SmartBear LoadUI - NeoLoad - Silk Peformer
42 Skills | Knowledge | Collaboration • Chrome developers tools • http://yslow.org/ • https://tools.pingdom.com/ • https://www.webpagetest.org/ • https://developers.google.com/speed/pag espeed/insights/ • https://www.sitespeed.io/ (automatically run using Docker) • Performance monitoring and management systems also can track timing metrics of the page load (RUM), like NewRelic Manual performance testing tools YSlow test example:
43 Skills | Knowledge | Collaboration 4. Implementation process
44 Skills | Knowledge | Collaboration Performance testing process activities 1. Planning 2. Implementing 3. Executing 4. Analyzing and reporting 5. Continuous Integration
45 Skills | Knowledge | Collaboration 1. Planning Identify Performance Acceptance Criteria and KPIs • Response time for different type of transactions -> a user concern • Throughput (tps/rps) -> a business concern • Resource utilization -> a system concern • Concurrent users number • Accepted Error rate • Accepted deviation for response time, resource utilization • System behavior when overloaded
46 Skills | Knowledge | Collaboration 1. Planning Performance requirement analysis Bad sample of requirements: 1. Response time should be no more than 4 seconds 2. System should be able to deal with 10 thousands concurrent users It is not clear whether: • Response time should be 4 seconds for all requests? • What if all response time will be 3.99 seconds? • What if most request will response with 2 seconds but several with 6 seconds? • What users do and how often? Open Main page or more? • What if more than 10 ths. users? System should scale, or new users will be rejected or response time just increases?
47 Skills | Knowledge | Collaboration 1. Planning Performance requirement analysis Good sample of requirements: 1. Set average response time for different types of transactions: • Simple navigation requests - 1 second • Logging - 2 seconds • Search and buy - 4 seconds 2. Set deviation - no more than 15% 3. Set failure rate - no more than 1% 4. Set CPU, memory, network, other server-metrics thresholds 5. Set response times, deviation, server-side metrics for different number of users - for 1k, 5k, 10k 6. System should reject new users in case of overload, showing informing message
48 Skills | Knowledge | Collaboration 1. Planning Identify the Test Environment • Identify the logical and physical production architecture for performance testing • Compare the both test and production environments while identifying the testing environment TEST environment MUST BE the same as PROD (or run tests on PROD) • Get resolve the environment-related concerns if any – using stabs for 3-rd parties or others • Analyze whether additional tools are required for performance testing, like monitoring tools. Install such tools. Identify scope of load testing (product parts, 3-rd party services in/out of scope) Identify technical nuances • Scheduling services? • Ping calls • Client’s internet connection speed? • Static content hosting: CDN or own servers? • Target region (USA, Europe, etc.)?
49 Skills | Knowledge | Collaboration 1. Planning Plan and Design Tests • Identify key usage scenario and workload (Load, Capacity, Spike, etc.) • Define test data • Establish metrics to be collected What transactions to include: 1. Critical transactions Example: System login, session support 2. Mostly used transactions by real users Example: System login/logout, main page 3. Business required Example: some specific feature 4. Risky transactions Example: Checkout, payment 5. Heavy transactions Example: File download/upload
50 Skills | Knowledge | Collaboration User Community Modeling Language (UCML) for Performance Test Workloads 1. Planning
51 Skills | Knowledge | Collaboration Prerequisites solving 1. Planning • Add your load generator machine IP(s) into whitelisting (if needed) • Request test users or create them by yourself • Request test data (products names, files, etc.) • Request 3-rd party dependent data (payment cards, etc.) Configure load-generation environment The output of this stage is prepared Performance Test Plan • Environment capacity (CPU, memory) should be enough to run required number of users during some time • Setup Master-Slave architecture in case of distributed load testing • Setup monitoring tools to track load server(s) health during tests running
52 Skills | Knowledge | Collaboration 2. Implementing • Develop performance scripts • Put assertions points and wait timers to make it a real time scenario • Run several smoke runs to calibrate scripts to the target environments in accordance with test design Simulate real users behavior basing on Production usage statistic
53 Skills | Knowledge | Collaboration 3. Executing It very depends on a project, but still we can suggest next How to choose and run correct load profiles
54 Skills | Knowledge | Collaboration 0. Warm up your product servers before load tests run - Run some short smoke load test for several users 3. Executing
55 Skills | Knowledge | Collaboration 1. Run several baseline Load tests to determine benchmark performance metrics levels: - virtual users: 5-100 - duration: 30-60m - rump-up – 10m; run – 20m; shutdown – 5m !!! Run each test at least 2 but better 3 times to be sure in results 3. Executing
56 Skills | Knowledge | Collaboration 2. Run the Capacity test profile and determine when Capacity point (system limit) happens: - virtual users: a lot, depends on the system - duration: 2-3 hours - rump-up – 2-3 hour; run – NA; shutdown – NA !!! Run each test at least 2 but better 3 times to be sure in results 3. Executing
57 Skills | Knowledge | Collaboration How to understand that you reach Capacity point: When transactions Response time increases sharply with load increase. Note: - although load increase, response time should not increase sharply. However, small deviation is acceptable; - response time also could start decrease dramatically, what means that you start receive responses with errors. When the Error rate increases with load increase Note: often, small error rate is acceptable (1-3%), especially during large load tests. Crash of the servers or one of them (web, application, DB, etc). 3. Executing
58 Skills | Knowledge | Collaboration Capacity point example V users: 10 ths. Status: failed Error rate: test stopped on 28%
59 Skills | Knowledge | Collaboration Capacity point example V users: 9 ths. Status: failed Error rate: test stopped on 49%
60 Skills | Knowledge | Collaboration Capacity point example V users: 8 ths. Status: passed Error rate: 0.21% (500 errors from 236 ths. transactions
61 Skills | Knowledge | Collaboration 3. Run the Load test profile with: - virtual users: 50-80% of Capacity point or/and required load number - duration: 2-3 hours - rump-up – 60m; run – 60m; shutdown – 20m !!! Run each test at least 2 but better 3 times to be sure in results 3. Executing
62 Skills | Knowledge | Collaboration 4. Run Component tests for each service / micro-service / function separately: - could be run before or after Endurance test, especially if issues for some service(s) were found - virtual users: firstly, like for Capacity test, then like for Load profile - duration: 1-2 hours - rump-up, run and shutdown timings depend on component test profile !!! Run each test at least 2 but better 3 times to be sure in results 3. Executing
63 Skills | Knowledge | Collaboration 5. Run the Endurance test profile: - virtual users: 20-40% of Capacity point - duration: 8-72 hours - rump-up – 30m; run – 23р; shutdown – 30m !!! Run each test at least 2 but better 3 times to be sure in results 3. Executing
64 Skills | Knowledge | Collaboration 6. Run other types of load testing and play with your scenarios combinations. - volume test, Spike, Reliability/Recovery, Volume tests - each of these tests could be run on the first stages, if it is prioritized be business - run load tests for different user flows combinations !!! Run each test at least 2 but better 3 times to be sure in results 3. Executing
65 Skills | Knowledge | Collaboration • In ongoing development -> Verifying and validating component, queue of components, and integration related performance & robustness • Before release -> Verifying and validating the whole product performance & robustness before release. • Maintenance -> Verifying and validating architectural, configurational, capacity-related, db-related, and integration-related changes When to perform 3. Executing
66 Skills | Knowledge | Collaboration • Collect and analyze load tests data • Investigate possible bottlenecks (memory, disk, processor, process, cache, network, etc.) resource usage like (memory, CPU, network, etc.) • Generate the Performance analysis reports Note: The report form is - a performance test summary report - transactions details, hardware utilization, etc. - the comparison of actual and expected KPIs • Based on the analysis prepare recommendation report • Share report with the team and stakeholders 4. Analyzing and reporting
67 Skills | Knowledge | Collaboration Step 9 – Report example
68 Skills | Knowledge | Collaboration 5. Continuous Integration or Development Continuous Integration Build. Unit test Deploy to Dev Server Integration Test Deploy to Test/QA Server Automated Functional Tests Deploy to Load Server Performance test in CI + monitoring Manual QA Test Promote to Stage / Pre-Prod Ideally, if you have separate environment for load testing Note: you can implement load tests on Stage too, if it is the same as Prod
69 Skills | Knowledge | Collaboration 5. Performance testing engineer skills
70 Skills | Knowledge | Collaboration Performance testing engineer skills Goodenough programmingskills (Python,Java,C++etc.) Loadtoolsandtheirsadvantages/disadvantages(JMeter, LoadRunner,Gatling etc.) RUM tools(NewRelic,AppDynamics,CloudWatch,Zabbixetc.) CI integrationtools Tools knowledge Knowledgeof performancemeasurementsandmetrics.Abilitytocalculate/interpretthem The client performancerequirements andgoals analysis Identifyingtransactionsandworkflows– calculatingworkloadTPSgoals andrates Performance theory knowledge
71 Skills | Knowledge | Collaboration Performance testing engineer skills Experiencein differentOSadministration:Windows,Linuxdistributives Interpretanddrawsystemsdiagrams Basicunderstandingof systemsenvironments– sharedresources,components,andservices Thedifferencesbetween productionandtestenvironments– containers,cloud, virtualization,andconfigurationmanagement RelationalandNoSQLdatabases ContinuousIntegrationandDeploymentconcepts Systems and architecture knowledge, DevOps
72 Skills | Knowledge | Collaboration Performance testing engineer skills Projectmanagementskills Communicationskills Eagerforconstantself-development andimprovement. Open fornew technologies Soft skills
73 Skills | Knowledge | Collaboration Useful Links About performancetesting: http://docplayer.net/29696161-Performance-matters-key-consumer-insights.html http://www.softwaretestingclass.com/what-is-performance-testing/ https://www.blazemeter.com/blog http://revolutionit.com.au/performance-testing-digital-projects-dont-leave-it-to-the-developers/ Concurrentuserscalculation: https://blog.xceptance.com/2013/07/26/concurrent-users-the-art-of-calculation/ https://techbeacon.com/how-many-virtual-users-do-i-need-load-testing https://www.webperformance.com/library/tutorials/CalculateNumberOfLoadtestUsers/ Browserperformance: https://www.slideshare.net/nicjansma/measuring-real-user-performance-in-the-browser Booksfor start: “Web Load Testing ForDummies”,Scott BarberwithColin Mason “JMeterCookbook”, Bayo Erinle
74 Skills | Knowledge | Collaboration

More Related Content

DOCX
Shriraam-performance test engineer 5.4 years
byshriraam ms
 
DOC
Saravanan_Performance_Test
bysaravanan subramani
 
DOC
Resume
byRajesh Reddy Badam
 
DOCX
Sameer_Resume
bySameer Mohammad
 
DOC
vignesh rajaraman (2)
byVicky Anu
 
DOCX
Moovendan M Resume
byMoovendan Munirathinam
 
DOCX
Resume_Gokul
byGokulakrishnan Andavar
 
DOC
UmasankarResume (1)
byGoverdhan Umasankar Bhamidipati
 
Shriraam-performance test engineer 5.4 years
byshriraam ms
 
Saravanan_Performance_Test
bysaravanan subramani
 
Resume
byRajesh Reddy Badam
 
Sameer_Resume
bySameer Mohammad
 
vignesh rajaraman (2)
byVicky Anu
 
Moovendan M Resume
byMoovendan Munirathinam
 
Resume_Gokul
byGokulakrishnan Andavar
 
UmasankarResume (1)
byGoverdhan Umasankar Bhamidipati
 

What's hot

DOC
Joseph Inbaraj S 11+ years ALM Admin Resume
byqtpjoseph
 
DOCX
Chirantan Chakraborty_Resume
bychirantan chakraborty
 
DOC
CV@Sapna@Performance Test Engineer
bySapna Sharma
 
PDF
Bottlenecks exposed web app db servers
byUpender Dravidum
 
DOC
Test Lead_Anshul_Varshney
byAnshul Varshney
 
DOC
7.3 Years of Experience
byKandadi Ragavendra Reddy
 
DOC
Pavan Patke - Performance Testing CV
byPavan Patke
 
DOC
Vibin chelladurai_Mainframe_CSC
byVibin Chelladurai
 
DOCX
Performance Test Consultant
byJEEVA VELLAISAMY
 
DOC
Nirmal kumar
byNirmal Kumar
 
DOC
Ananthavalli_Resume
byAnanthavalli Gopalakrishnan
 
DOC
Dhananjayulu Neeli_CV
byDHANANJAYULU NEELI
 
DOC
Ram pega cssa resume
byAshock Roy
 
DOC
7 Years of Experience
byKandadi Ragavendra Reddy
 
DOC
Vishal_Project Lead
byvishalreddy alty
 
DOC
Md Rafi_Resume
byMahammad Rafi Shaik
 
PPTX
Web 2.0 Development with IBM DB2
byVladimir Bacvanski, PhD
 
DOC
Sathish_Nugehalli_
bySatish Nugehalli
 
DOCX
Updated resume
byshriraam ms
 
DOC
RESUME PEGA
byKOTTE PRAKASH
 
Joseph Inbaraj S 11+ years ALM Admin Resume
byqtpjoseph
 
Chirantan Chakraborty_Resume
bychirantan chakraborty
 
CV@Sapna@Performance Test Engineer
bySapna Sharma
 
Bottlenecks exposed web app db servers
byUpender Dravidum
 
Test Lead_Anshul_Varshney
byAnshul Varshney
 
7.3 Years of Experience
byKandadi Ragavendra Reddy
 
Pavan Patke - Performance Testing CV
byPavan Patke
 
Vibin chelladurai_Mainframe_CSC
byVibin Chelladurai
 
Performance Test Consultant
byJEEVA VELLAISAMY
 
Nirmal kumar
byNirmal Kumar
 
Ananthavalli_Resume
byAnanthavalli Gopalakrishnan
 
Dhananjayulu Neeli_CV
byDHANANJAYULU NEELI
 
Ram pega cssa resume
byAshock Roy
 
7 Years of Experience
byKandadi Ragavendra Reddy
 
Vishal_Project Lead
byvishalreddy alty
 
Md Rafi_Resume
byMahammad Rafi Shaik
 
Web 2.0 Development with IBM DB2
byVladimir Bacvanski, PhD
 
Sathish_Nugehalli_
bySatish Nugehalli
 
Updated resume
byshriraam ms
 
RESUME PEGA
byKOTTE PRAKASH
 

Similar to Микола Ковш “Performance Testing Implementation From Scratch. Why? When and How?”

PPTX
Performance Testing from Scratch + JMeter intro
byMykola Kovsh
 
PPTX
Performance Testing
bySelin Gungor
 
PDF
Adding Performance Testing to a Software Development Project
byCris Holdorph
 
PDF
Need for Speed: How to Performance Test the right way by Annie Bhaumik
byQA or the Highway
 
PDF
Comprehensive Performance Testing: From Early Dev to Live Production
byTechWell
 
PPTX
Performance testing
byChalana Kahandawala
 
PPT
Performance Testing Overview
byJames Venetsanakos
 
PDF
Heuristics of performance testing
byPradeep Soundararajan
 
PDF
Scalability and performance for e commerce
byEcommerce Solution Provider SysIQ
 
PPT
JMeter
byMd Samsul Kabir
 
PPT
Performance testing and rpt
bySivaprasanthRentala1975
 
PPTX
An Introduction to Performance Testing
bySWAAM Tech
 
PPTX
Introduction to Performance Testing
byTharinda Liyanage
 
PDF
Web Performance Analysis - TCF Pro 2009
byGuy Ferraiolo
 
PDF
Fundamentals Performance Testing
byBhuvaneswari Subramani
 
PPT
performance testing
byShyaamini Balu
 
PPT
PERFTEST.ppt
byhemanthKumar954692
 
PPT
PERFTEST.ppt
byMeghanaAkkapalli
 
PPTX
Introduction to performance testing
byRichard Bishop
 
PPTX
Performance testing
byHassan Mohammed
 
Performance Testing from Scratch + JMeter intro
byMykola Kovsh
 
Performance Testing
bySelin Gungor
 
Adding Performance Testing to a Software Development Project
byCris Holdorph
 
Need for Speed: How to Performance Test the right way by Annie Bhaumik
byQA or the Highway
 
Comprehensive Performance Testing: From Early Dev to Live Production
byTechWell
 
Performance testing
byChalana Kahandawala
 
Performance Testing Overview
byJames Venetsanakos
 
Heuristics of performance testing
byPradeep Soundararajan
 
Scalability and performance for e commerce
byEcommerce Solution Provider SysIQ
 
JMeter
byMd Samsul Kabir
 
Performance testing and rpt
bySivaprasanthRentala1975
 
An Introduction to Performance Testing
bySWAAM Tech
 
Introduction to Performance Testing
byTharinda Liyanage
 
Web Performance Analysis - TCF Pro 2009
byGuy Ferraiolo
 
Fundamentals Performance Testing
byBhuvaneswari Subramani
 
performance testing
byShyaamini Balu
 
PERFTEST.ppt
byhemanthKumar954692
 
PERFTEST.ppt
byMeghanaAkkapalli
 
Introduction to performance testing
byRichard Bishop
 
Performance testing
byHassan Mohammed
 

More from Dakiry

PDF
НАРЦИСИЗМ ЯК ПАСИВНЕ КУРІННЯ
byDakiry
 
PDF
МАНІПУЛЯЦІЇ: ХТО КОГО І ДЛЯ ЧОГО? - Інна Тіторенко
byDakiry
 
PPTX
How to run a discovery workshop
byDakiry
 
PPTX
З понеділка йду на новий проект. The tester’s version - Олександра Зубаль
byDakiry
 
PDF
Робота з текстом: від чернетки до опублікування
byDakiry
 
PPTX
Контентна стратегія в ІТ: від статті до першого ліда
byDakiry
 
PPTX
Oleh Shpyrna "Security Testing Basics: Check your Webapp for gaps before l_unch"
byDakiry
 
PPTX
Stepan Shykerynets "Power of QA (A Journey: From Hell to Heaven. Story of gr...
byDakiry
 
PDF
Микола Солопій "Selenium рулить, однак..."
byDakiry
 
PDF
Oleksandra Zubal "Project starters: test automation view"
byDakiry
 
PDF
Vladyslav Romanchenko "How to keep high code quality without e2e tests"
byDakiry
 
PPTX
Діана Пінчук "Як відрізнити авторизацію від аутентифікації та перестати бояти...
byDakiry
 
PPT
Yuriy Malyi "E2E testing organization in multi-system projects"
byDakiry
 
PPTX
Petro Tarasenko "You've become a TL. What's next?"
byDakiry
 
PDF
Roman Yakymchuk "Дослідницьке тестування. Перезапуск"
byDakiry
 
PPTX
Maryna Shulga "Mission Impossible. Впровадити тест процеси, якщо ніхто цього ...
byDakiry
 
PDF
Олексій Брошков "Мистецтво Дослідницького Тестування"
byDakiry
 
PPSX
Альона Тудан " Життя QA в ажурі"
byDakiry
 
PPTX
Андрій Степура "Тренди в публічних виступах"
byDakiry
 
PPTX
Зоряна Борбулевич "Підхід, який трансформував компанію Microsoft: ННК і його...
byDakiry
 
НАРЦИСИЗМ ЯК ПАСИВНЕ КУРІННЯ
byDakiry
 
МАНІПУЛЯЦІЇ: ХТО КОГО І ДЛЯ ЧОГО? - Інна Тіторенко
byDakiry
 
How to run a discovery workshop
byDakiry
 
З понеділка йду на новий проект. The tester’s version - Олександра Зубаль
byDakiry
 
Робота з текстом: від чернетки до опублікування
byDakiry
 
Контентна стратегія в ІТ: від статті до першого ліда
byDakiry
 
Oleh Shpyrna "Security Testing Basics: Check your Webapp for gaps before l_unch"
byDakiry
 
Stepan Shykerynets "Power of QA (A Journey: From Hell to Heaven. Story of gr...
byDakiry
 
Микола Солопій "Selenium рулить, однак..."
byDakiry
 
Oleksandra Zubal "Project starters: test automation view"
byDakiry
 
Vladyslav Romanchenko "How to keep high code quality without e2e tests"
byDakiry
 
Діана Пінчук "Як відрізнити авторизацію від аутентифікації та перестати бояти...
byDakiry
 
Yuriy Malyi "E2E testing organization in multi-system projects"
byDakiry
 
Petro Tarasenko "You've become a TL. What's next?"
byDakiry
 
Roman Yakymchuk "Дослідницьке тестування. Перезапуск"
byDakiry
 
Maryna Shulga "Mission Impossible. Впровадити тест процеси, якщо ніхто цього ...
byDakiry
 
Олексій Брошков "Мистецтво Дослідницького Тестування"
byDakiry
 
Альона Тудан " Життя QA в ажурі"
byDakiry
 
Андрій Степура "Тренди в публічних виступах"
byDakiry
 
Зоряна Борбулевич "Підхід, який трансформував компанію Microsoft: ННК і його...
byDakiry
 

Recently uploaded

DOCX
Best Place to Learn Old or Aged Verified Square Accounts in WWW.docx
byTopSelleriT
 
PDF
Bohdan Haidabrus: Lessons Learned from Scaling Agile: A Case Study of SAFe Im...
byLviv Startup Club
 
PDF
How to Buy Old Gmail Accounts in Bulk Safely in the USA.pdf
bypvaisback is Online Marketing site
 
PDF
Ultra Map Section II Multifaceted Second Edition.pdf
byBrij Consulting, LLC
 
PDF
AnyDesk, The Leading Remote Desktop Software | AnyDesk, o software líder em a...
byTechMeetups
 
DOCX
Buy Instagram Account Purchase_ A Safe and Secure Approach.docx
bybarbeymcdanielokczz
 
PDF
𝐖𝐚𝐠 𝐂𝐥𝐨𝐧𝐞 𝐏𝐞𝐭 𝐂𝐚𝐫𝐞 𝐀𝐩𝐩 𝐃𝐞𝐯𝐞𝐥𝐨𝐩𝐦𝐞𝐧𝐭
byV3CUBE TECHNOLABS
 
DOCX
Important Asset To know About Buying Old or Aged Verified Bybit Accounts.docx
byTopSelleriTDotCom
 
PDF
Guide to Safely Buying Gmail Accounts for Safely-Aged.pdf
bysmartusapva.com
 
PDF
Top Platform to Buy Facebook Accounts in Bulk Safely.pdf
byBuy Facebook Accounts
 
PDF
How To B,u,y Verified Cash App Accounts – Understanding Safe Usage, BTC Featu...
byapasanilion108
 
PDF
rf_mccaffrey_stocksforthelongrun_revisited_online.v2 (1).pdf
byHenry Tapper
 
PDF
Prasenjit Bhaumik Plano - Proficient In JavaScript, Python, And Database Mana...
byPrasenjit Bhaumik Plano
 
PDF
AI,Analytics, Digital HR, Agentic AI.pdf
bypgp25chinica
 
PDF
LA Building Inspections & Compliance offers
byoliversmith20262019
 
DOCX
Buy Verified Cash App Accounts In The Us
bysebastianparsonst8
 
PDF
Andria Sergio - Known For Her Accuracy And Professionalism
byAndria Sergio
 
PDF
How to Buy Verified Bybit Accounts in 2026.pdf
bymarketing
 
PDF
Best Place to Learn Old or Aged Verified Paysera Accounts in WWW.pdf
byTopSelleriT
 
DOCX
What Are Telegram Accounts and Why Are They Important for Global Communicatio...
byx24gm4qs23vusc8s2b53
 
Best Place to Learn Old or Aged Verified Square Accounts in WWW.docx
byTopSelleriT
 
Bohdan Haidabrus: Lessons Learned from Scaling Agile: A Case Study of SAFe Im...
byLviv Startup Club
 
How to Buy Old Gmail Accounts in Bulk Safely in the USA.pdf
bypvaisback is Online Marketing site
 
Ultra Map Section II Multifaceted Second Edition.pdf
byBrij Consulting, LLC
 
AnyDesk, The Leading Remote Desktop Software | AnyDesk, o software líder em a...
byTechMeetups
 
Buy Instagram Account Purchase_ A Safe and Secure Approach.docx
bybarbeymcdanielokczz
 
𝐖𝐚𝐠 𝐂𝐥𝐨𝐧𝐞 𝐏𝐞𝐭 𝐂𝐚𝐫𝐞 𝐀𝐩𝐩 𝐃𝐞𝐯𝐞𝐥𝐨𝐩𝐦𝐞𝐧𝐭
byV3CUBE TECHNOLABS
 
Important Asset To know About Buying Old or Aged Verified Bybit Accounts.docx
byTopSelleriTDotCom
 
Guide to Safely Buying Gmail Accounts for Safely-Aged.pdf
bysmartusapva.com
 
Top Platform to Buy Facebook Accounts in Bulk Safely.pdf
byBuy Facebook Accounts
 
How To B,u,y Verified Cash App Accounts – Understanding Safe Usage, BTC Featu...
byapasanilion108
 
rf_mccaffrey_stocksforthelongrun_revisited_online.v2 (1).pdf
byHenry Tapper
 
Prasenjit Bhaumik Plano - Proficient In JavaScript, Python, And Database Mana...
byPrasenjit Bhaumik Plano
 
AI,Analytics, Digital HR, Agentic AI.pdf
bypgp25chinica
 
LA Building Inspections & Compliance offers
byoliversmith20262019
 
Buy Verified Cash App Accounts In The Us
bysebastianparsonst8
 
Andria Sergio - Known For Her Accuracy And Professionalism
byAndria Sergio
 
How to Buy Verified Bybit Accounts in 2026.pdf
bymarketing
 
Best Place to Learn Old or Aged Verified Paysera Accounts in WWW.pdf
byTopSelleriT
 
What Are Telegram Accounts and Why Are They Important for Global Communicatio...
byx24gm4qs23vusc8s2b53
 

Микола Ковш “Performance Testing Implementation From Scratch. Why? When and How?”

  • 1.
    1 Skills | Knowledge| CollaborationSkills | Knowledge | Collaboration PerformanceTestingImplementationFromScratch.Why? WhenandHow? by Mykola Kovsh
  • 2.
    2 Skills | Knowledge| Collaboration Speaker info Hi, I’m Mykola Kovsh PerformanceTestEngineer PerformanceTesting Departmentat Testing Centerof Excellence, Ciklum, Ukraine •4 yearsin IT •Performance/ Automation/ Manualtesting www.linkedin.com/in/mykolakovsh/
  • 3.
    3 Skills | Knowledge| Collaboration Content Performance testing engineer skills5 Why performance testing is important?1 Introduction to performance testing. Load profiles, metrics2 Implementation process4 Performance testing tools3 Links and Q&A session
  • 4.
    4 Skills | Knowledge| Collaboration 1. Why performance testing is important?
  • 5.
    5 Skills | Knowledge| Collaboration Why performance testing is important ✓ 50% of frustrated users will visit another website to accomplish their activity and 22% won't return. ✓ 49% of respondents expect web pages to load in under 2 seconds. ✓ 30% expect a 1-second response. ✓ 18% expect a site to load immediately.* * http://docplayer.net/29696161-Performance-matters-key-consumer-insights.html
  • 6.
    6 Skills | Knowledge| Collaboration From user perception point of view What is a fast web-side? < 1 second very fast < 2 seconds quite fast 2-4 seconds acceptable 5-15 seconds slow >15 seconds too slow * https://www.hobo-web.co.uk/your-website-design-should-load-in-4-seconds/
  • 7.
    7 Skills | Knowledge| Collaboration
  • 8.
    8 Skills | Knowledge| Collaboration How smartphone users react to slow web-sites Curse at their phone, 23% Scream at their phone, 11% Throw their phone, 4% Behave more or less normally, 62%
  • 9.
    9 Skills | Knowledge| Collaboration Cost of poor performance * http://www.webperformancetoday.com/2010/06/15/everything-you-wanted-to-know-about-web-performance/  If your average sales per hour is $5,000 1 minute of downtime is costing over $80 then
  • 10.
    10 Skills | Knowledge| Collaboration Cost of poor performance  A 1-second page load delay equals*: * http://www.webperformancetoday.com/2010/06/15/everything-you-wanted-to-know-about-web-performance/ 16 % decrease in customer satisfaction 11 % fewer page views 7 % loss in conversions
  • 11.
    11 Skills | Knowledge| Collaboration Cost of poor performance For example: ✓ John Lewis’s website went down around 3.20 pm on Black Friday - analysts estimated it could cost £ 2.8m pounds. ✓ Web giant like Amazon would lose as much as $120 000 per minute of downtime. ✓ Microsoft Bing found that a two-second slowdown caused a 4.3 percent reduction in revenue per user. ✓ Website Shopzilla reduced page load times from 7 seconds to 2 seconds, resulting in a 7–12 percent increase in revenue and 50 percent reduction in hardware costs. * http://www.webperformancetoday.com/2010/06/15/everything-you-wanted-to-know-about-web-performance/
  • 12.
    12 Skills | Knowledge| Collaboration When performance testing is needed ✓ The solution is already struggling from performance problems. ✓ The solution have to deal with big amount of concurrent users presently or in future (~ 300-500 concurrent users and higher). ✓ The solution have a large database or should transfer or process big amount of data in real time (~ 10 and more concurrent users). ✓ The solution has complex architecture and a lot of internal and external integrations running concurrently.
  • 13.
    13 Skills | Knowledge| Collaboration 2. Introduction to Performance Testing
  • 14.
    14 Skills | Knowledge| Collaboration ✓ Performance Testing - the process of testing to determine the performance of a software product (ISTQB Foundation). ✓ Performance testing – is a non-functional software testing technique which determines responsiveness, stability, reliability and resource usage of system under a certain user load (Wikipedia). ✓ Web load testing is nothing more than exercising a website under a variety of production-like conditions to determine how it’s going to work and to identify (and hopefully resolve) problems before your customers find them (WebLoadTestingForDummies). Performance testing. Definition
  • 15.
    15 Skills | Knowledge| Collaboration Determine responsiveness, stability, reliability and resource usage of system under a certain user load Demonstrate that the system meets performance criteria (KPIs) Compare different system configurations and versions to evaluate performance improvement/degradation. Performance trends tracking during the time Determine system behavior under different load Evaluate the system capacity Scalability. Determine ability of a system to handle a growing amount of workload Prepare the application for planned load (e-commerce: Black Friday, Marketing Campaign. Finance: quarter/annual reporting, etc.) Find which components of the system perform poorly under certain workload Goals of performance testing:
  • 16.
    16 Skills | Knowledge| Collaboration Slow sub-systems / functions (poor response) Low capacity point Configuration problems (web-server, load balancers, db etc) Dead-lock while simultaneous load Flawed queue logic Incorrect synchronization of recourses Database issues e.g. size, indexing, replication Memory, space and connections leaks Poor network configuration CPU, Memory utilization Functionality bugs (how system should behave under overload, others). Performance bottlenecks:
  • 17.
    17 Skills | Knowledge| Collaboration Functional & Performance testing comparison # Functional testing Performance testing 1 To verify the accuracy of the system against expected results To verify the behavior of the system at various load conditions 2 Manual or automated Automated only 3 Could be done without special tools Special set of tools is used including analyzing and monitoring ones 4 One user performing all operations Several users performing desired operations 5 Involvement required from Customer, Tester and Developer Involvement required from Customers, Tester, Developer, DB admins, DevOps 6 Test environment capacity/size could differ from Production Requires close to Production Test environment!!! based on: http://www.softwaretestinghelp.com/introduction-to-performance-testing-loadrunner-training-tutorial-part-1/
  • 18.
    18 Skills | Knowledge| Collaboration Performance testing Approaches Protocol-level Load Generation (HTTP, Web-sockets, TCP, JDBC, etc.) Device level Load Generation (User Interface)
  • 19.
    19 Skills | Knowledge| Collaboration Performance testing Approaches Protocol-level Load Generation (HTTP, Web-sockets, TCP, JDBC, etc.) Device level Load Generation (User Interface)
  • 20.
    20 Skills | Knowledge| Collaboration Performance testing Targets Server-side performance (web, app, database) Client-side (manual) performance Network performance Device performance (hardware: battery consumption, CPU, memory, etc.)
  • 21.
    21 Skills | Knowledge| Collaboration Performance testing Targets Server-side performance (web, app, database) Client-side (manual) performance Network performance Device performance (hardware: battery consumption, CPU, memory, etc.)
  • 22.
    22 Skills | Knowledge| Collaboration Server-side performance
  • 23.
    23 Skills | Knowledge| Collaboration Levels of server-side performance testing • End-to-End load testing • Integration Load testing • Component/API Load testing • DB performance testing
  • 24.
    24 Skills | Knowledge| Collaboration Types of Performance Testing (load profiles) Stress/Capacity test Max Designed Operation Capacity Volume test 8-72 hours or longer + Component Test + Reliability / Recovery Test Server-side performance
  • 25.
    25 Skills | Knowledge| Collaboration Client-side (manual) Performance
  • 26.
    26 Skills | Knowledge| Collaboration Client-side performance testing - testing of one separate page load from client/browser side Client-side Performance
  • 27.
    27 Skills | Knowledge| Collaboration More detailed process of page load Client-side Performance
  • 28.
    28 Skills | Knowledge| Collaboration First thing the user sees Client-side Performance
  • 29.
    29 Skills | Knowledge| Collaboration Visual Experience Client-side Performance 1. First Paint 2. First Contentful Paint 3. First Meaningful Paint 4. Visually Complete based on: https://www.slideshare.net/nicjansma/measuring-real-user-performance-in-the-browser
  • 30.
    30 Skills | Knowledge| Collaboration Client-side Performance Chrome performance testing
  • 31.
    31 Skills | Knowledge| Collaboration 1. Run client-side performance of the page when there is 0 load. 2. Set performance metrics baselines based on step 1 3. Run client-side performance when there is different load on the server – during Capacity/Load/Spike etc. tests 4. Compare results with baselines, analyze and summarize possible issues Flow we suggest: Client-side Performance
  • 32.
    32 Skills | Knowledge| Collaboration Load testing metrics
  • 33.
    33 Skills | Knowledge| Collaboration Load testing metrics (synthetic monitoring) Application-side Server-side - Response time - Throughput (rps/tps/tpm) - Concurrent users - Error rate (response code) - Number of transactions passed/failed - Network traffic - CPU - Memory - Network - Disk - DB connections - Logs error, warnings
  • 34.
    34 Skills | Knowledge| Collaboration Response time = Latency (travelling across a network) + Processing time (system processing of request) • Average response time • Peak response time (max) • Response time with 95% or other percentile Response time Application-side metrics
  • 35.
    35 Skills | Knowledge| Collaboration Throughput - how many simultaneous requests/transactions per second/minute application can handle Note:  TPS could correlate with response time if requests are consequent. The longer response – the lower tps  TPS does not correlate directly with response time if requests are parallel. consequent > 1 request – 1 sec response – 1 tps parallel > 10 requests – 1 sec response – 10 tps  TPS could be improved by improving response time or by increasing concurrent users Throughput (rps/tps/tpm) Application-side metrics
  • 36.
    36 Skills | Knowledge| Collaboration Network traffic, Response codes, Error rate Application-side metrics Network traffic – shows how much data is flowing back and forth from your servers (Kbytes or Mbytes / sec). We can compare this metric to the response-time metric to see how the throughput affects transaction performance. Error rate – is the mathematical calculation that produces a percentage of problem requests compared to all requests. It is no standard for tolerable error rate. Some projects consider 1% error rate successful in case the system can handle maximum load without crash. Others consider any errors. Still, few errors is not uncommon, especially for large load.
  • 37.
    37 Skills | Knowledge| Collaboration • You can monitor server-side metrics directly on the server (Linux, Windows) • You can automate this process creating some monitoring agent to track metrics • You can use one of monitoring tools: - CloudWatch (Amazon) - AppDynamics - DynaTrace - NewRelic - Graylog, etc. Server-side monitoring tools *In addition to the server metrics monitoring during the load test (synthetic monitoring), monitoring tools allow Real User Monitoring (RUM). RUM is a type of performance monitoring that captures and analyzes each transaction by real users of a website or application. Unlike synthetic monitoring, RUM never rests. It collects data from each user using every browser across each request.
  • 38.
    38 Skills | Knowledge| Collaboration 3. Performance testing tools
  • 39.
    39 Skills | Knowledge| Collaboration Strong load testing tools should be able Traffic recorder Have IDE (console or GUI) which allows: a) Create Requests of required protocol (HTTP, HTTPs, WS, WSS, JDBC, TCP, AJAX, etc.) b) Support Transactions – to track time for all static data loading / redirections c) Create Load Scenarios with ability of parametrization d) Build different Load Profiles with rump-up and shut-down e) Have debugger Load runner engine Distributed testing Load test data saving (distributed), including client and server-side metrics Load test data monitoring in real time
  • 40.
    40 Skills | Knowledge| Collaboration - JMeter - Gatling - Locust - The Grinder - Apachebench - Artillery - Tsung - Vegeta - Siege - Boom - Wrk Open source load testing tools According to Load Impact tools comparison research*:  JMeter, Gatling, Grinder, Tsung and Boom all offer good performance, accuracy and reliability  Artillery, Locust and Siege have various issues with performance, accuracy and/or reliability  Performance-wise, Wrk and Apachebench are in a class of their own  NOTE: None of the tools tested can simulate thousands of VUs on a single machine without significant degradation in measurement accuracy * http://blog.loadimpact.com/open-source-load-testing-tool-benchmarks
  • 41.
    41 Skills | Knowledge| Collaboration Cloud-based tools: - BlazeMeter - LoadImpact - Loadstorm - Loader.io - Loadfocus, etc Commercial load tools Commercial Combiners: - HP Loadrunner - Microsoft TFS - SmartBear LoadUI - NeoLoad - Silk Peformer
  • 42.
    42 Skills | Knowledge| Collaboration • Chrome developers tools • http://yslow.org/ • https://tools.pingdom.com/ • https://www.webpagetest.org/ • https://developers.google.com/speed/pag espeed/insights/ • https://www.sitespeed.io/ (automatically run using Docker) • Performance monitoring and management systems also can track timing metrics of the page load (RUM), like NewRelic Manual performance testing tools YSlow test example:
  • 43.
    43 Skills | Knowledge| Collaboration 4. Implementation process
  • 44.
    44 Skills | Knowledge| Collaboration Performance testing process activities 1. Planning 2. Implementing 3. Executing 4. Analyzing and reporting 5. Continuous Integration
  • 45.
    45 Skills | Knowledge| Collaboration 1. Planning Identify Performance Acceptance Criteria and KPIs • Response time for different type of transactions -> a user concern • Throughput (tps/rps) -> a business concern • Resource utilization -> a system concern • Concurrent users number • Accepted Error rate • Accepted deviation for response time, resource utilization • System behavior when overloaded
  • 46.
    46 Skills | Knowledge| Collaboration 1. Planning Performance requirement analysis Bad sample of requirements: 1. Response time should be no more than 4 seconds 2. System should be able to deal with 10 thousands concurrent users It is not clear whether: • Response time should be 4 seconds for all requests? • What if all response time will be 3.99 seconds? • What if most request will response with 2 seconds but several with 6 seconds? • What users do and how often? Open Main page or more? • What if more than 10 ths. users? System should scale, or new users will be rejected or response time just increases?
  • 47.
    47 Skills | Knowledge| Collaboration 1. Planning Performance requirement analysis Good sample of requirements: 1. Set average response time for different types of transactions: • Simple navigation requests - 1 second • Logging - 2 seconds • Search and buy - 4 seconds 2. Set deviation - no more than 15% 3. Set failure rate - no more than 1% 4. Set CPU, memory, network, other server-metrics thresholds 5. Set response times, deviation, server-side metrics for different number of users - for 1k, 5k, 10k 6. System should reject new users in case of overload, showing informing message
  • 48.
    48 Skills | Knowledge| Collaboration 1. Planning Identify the Test Environment • Identify the logical and physical production architecture for performance testing • Compare the both test and production environments while identifying the testing environment TEST environment MUST BE the same as PROD (or run tests on PROD) • Get resolve the environment-related concerns if any – using stabs for 3-rd parties or others • Analyze whether additional tools are required for performance testing, like monitoring tools. Install such tools. Identify scope of load testing (product parts, 3-rd party services in/out of scope) Identify technical nuances • Scheduling services? • Ping calls • Client’s internet connection speed? • Static content hosting: CDN or own servers? • Target region (USA, Europe, etc.)?
  • 49.
    49 Skills | Knowledge| Collaboration 1. Planning Plan and Design Tests • Identify key usage scenario and workload (Load, Capacity, Spike, etc.) • Define test data • Establish metrics to be collected What transactions to include: 1. Critical transactions Example: System login, session support 2. Mostly used transactions by real users Example: System login/logout, main page 3. Business required Example: some specific feature 4. Risky transactions Example: Checkout, payment 5. Heavy transactions Example: File download/upload
  • 50.
    50 Skills | Knowledge| Collaboration User Community Modeling Language (UCML) for Performance Test Workloads 1. Planning
  • 51.
    51 Skills | Knowledge| Collaboration Prerequisites solving 1. Planning • Add your load generator machine IP(s) into whitelisting (if needed) • Request test users or create them by yourself • Request test data (products names, files, etc.) • Request 3-rd party dependent data (payment cards, etc.) Configure load-generation environment The output of this stage is prepared Performance Test Plan • Environment capacity (CPU, memory) should be enough to run required number of users during some time • Setup Master-Slave architecture in case of distributed load testing • Setup monitoring tools to track load server(s) health during tests running
  • 52.
    52 Skills | Knowledge| Collaboration 2. Implementing • Develop performance scripts • Put assertions points and wait timers to make it a real time scenario • Run several smoke runs to calibrate scripts to the target environments in accordance with test design Simulate real users behavior basing on Production usage statistic
  • 53.
    53 Skills | Knowledge| Collaboration 3. Executing It very depends on a project, but still we can suggest next How to choose and run correct load profiles
  • 54.
    54 Skills | Knowledge| Collaboration 0. Warm up your product servers before load tests run - Run some short smoke load test for several users 3. Executing
  • 55.
    55 Skills | Knowledge| Collaboration 1. Run several baseline Load tests to determine benchmark performance metrics levels: - virtual users: 5-100 - duration: 30-60m - rump-up – 10m; run – 20m; shutdown – 5m !!! Run each test at least 2 but better 3 times to be sure in results 3. Executing
  • 56.
    56 Skills | Knowledge| Collaboration 2. Run the Capacity test profile and determine when Capacity point (system limit) happens: - virtual users: a lot, depends on the system - duration: 2-3 hours - rump-up – 2-3 hour; run – NA; shutdown – NA !!! Run each test at least 2 but better 3 times to be sure in results 3. Executing
  • 57.
    57 Skills | Knowledge| Collaboration How to understand that you reach Capacity point: When transactions Response time increases sharply with load increase. Note: - although load increase, response time should not increase sharply. However, small deviation is acceptable; - response time also could start decrease dramatically, what means that you start receive responses with errors. When the Error rate increases with load increase Note: often, small error rate is acceptable (1-3%), especially during large load tests. Crash of the servers or one of them (web, application, DB, etc). 3. Executing
  • 58.
    58 Skills | Knowledge| Collaboration Capacity point example V users: 10 ths. Status: failed Error rate: test stopped on 28%
  • 59.
    59 Skills | Knowledge| Collaboration Capacity point example V users: 9 ths. Status: failed Error rate: test stopped on 49%
  • 60.
    60 Skills | Knowledge| Collaboration Capacity point example V users: 8 ths. Status: passed Error rate: 0.21% (500 errors from 236 ths. transactions
  • 61.
    61 Skills | Knowledge| Collaboration 3. Run the Load test profile with: - virtual users: 50-80% of Capacity point or/and required load number - duration: 2-3 hours - rump-up – 60m; run – 60m; shutdown – 20m !!! Run each test at least 2 but better 3 times to be sure in results 3. Executing
  • 62.
    62 Skills | Knowledge| Collaboration 4. Run Component tests for each service / micro-service / function separately: - could be run before or after Endurance test, especially if issues for some service(s) were found - virtual users: firstly, like for Capacity test, then like for Load profile - duration: 1-2 hours - rump-up, run and shutdown timings depend on component test profile !!! Run each test at least 2 but better 3 times to be sure in results 3. Executing
  • 63.
    63 Skills | Knowledge| Collaboration 5. Run the Endurance test profile: - virtual users: 20-40% of Capacity point - duration: 8-72 hours - rump-up – 30m; run – 23р; shutdown – 30m !!! Run each test at least 2 but better 3 times to be sure in results 3. Executing
  • 64.
    64 Skills | Knowledge| Collaboration 6. Run other types of load testing and play with your scenarios combinations. - volume test, Spike, Reliability/Recovery, Volume tests - each of these tests could be run on the first stages, if it is prioritized be business - run load tests for different user flows combinations !!! Run each test at least 2 but better 3 times to be sure in results 3. Executing
  • 65.
    65 Skills | Knowledge| Collaboration • In ongoing development -> Verifying and validating component, queue of components, and integration related performance & robustness • Before release -> Verifying and validating the whole product performance & robustness before release. • Maintenance -> Verifying and validating architectural, configurational, capacity-related, db-related, and integration-related changes When to perform 3. Executing
  • 66.
    66 Skills | Knowledge| Collaboration • Collect and analyze load tests data • Investigate possible bottlenecks (memory, disk, processor, process, cache, network, etc.) resource usage like (memory, CPU, network, etc.) • Generate the Performance analysis reports Note: The report form is - a performance test summary report - transactions details, hardware utilization, etc. - the comparison of actual and expected KPIs • Based on the analysis prepare recommendation report • Share report with the team and stakeholders 4. Analyzing and reporting
  • 67.
    67 Skills | Knowledge| Collaboration Step 9 – Report example
  • 68.
    68 Skills | Knowledge| Collaboration 5. Continuous Integration or Development Continuous Integration Build. Unit test Deploy to Dev Server Integration Test Deploy to Test/QA Server Automated Functional Tests Deploy to Load Server Performance test in CI + monitoring Manual QA Test Promote to Stage / Pre-Prod Ideally, if you have separate environment for load testing Note: you can implement load tests on Stage too, if it is the same as Prod
  • 69.
    69 Skills | Knowledge| Collaboration 5. Performance testing engineer skills
  • 70.
    70 Skills | Knowledge| Collaboration Performance testing engineer skills Goodenough programmingskills (Python,Java,C++etc.) Loadtoolsandtheirsadvantages/disadvantages(JMeter, LoadRunner,Gatling etc.) RUM tools(NewRelic,AppDynamics,CloudWatch,Zabbixetc.) CI integrationtools Tools knowledge Knowledgeof performancemeasurementsandmetrics.Abilitytocalculate/interpretthem The client performancerequirements andgoals analysis Identifyingtransactionsandworkflows– calculatingworkloadTPSgoals andrates Performance theory knowledge
  • 71.
    71 Skills | Knowledge| Collaboration Performance testing engineer skills Experiencein differentOSadministration:Windows,Linuxdistributives Interpretanddrawsystemsdiagrams Basicunderstandingof systemsenvironments– sharedresources,components,andservices Thedifferencesbetween productionandtestenvironments– containers,cloud, virtualization,andconfigurationmanagement RelationalandNoSQLdatabases ContinuousIntegrationandDeploymentconcepts Systems and architecture knowledge, DevOps
  • 72.
    72 Skills | Knowledge| Collaboration Performance testing engineer skills Projectmanagementskills Communicationskills Eagerforconstantself-development andimprovement. Open fornew technologies Soft skills
  • 73.
    73 Skills | Knowledge| Collaboration Useful Links About performancetesting: http://docplayer.net/29696161-Performance-matters-key-consumer-insights.html http://www.softwaretestingclass.com/what-is-performance-testing/ https://www.blazemeter.com/blog http://revolutionit.com.au/performance-testing-digital-projects-dont-leave-it-to-the-developers/ Concurrentuserscalculation: https://blog.xceptance.com/2013/07/26/concurrent-users-the-art-of-calculation/ https://techbeacon.com/how-many-virtual-users-do-i-need-load-testing https://www.webperformance.com/library/tutorials/CalculateNumberOfLoadtestUsers/ Browserperformance: https://www.slideshare.net/nicjansma/measuring-real-user-performance-in-the-browser Booksfor start: “Web Load Testing ForDummies”,Scott BarberwithColin Mason “JMeterCookbook”, Bayo Erinle
  • 74.
    74 Skills | Knowledge| Collaboration

Editor's Notes

  • #13 AppsFlyer – 12-18 mln requests per second
  • #24 Highlight a value, combination of benefits and approaches
  • #25 - Endurance test – тестирование стабильносты. Профиль тот же что при лоад, но время дольше. При пиках интересно сможет ли система восставновится после пика, освободить ресурсы и т.д. Пики можно делать только для какого-то сценария, либо для всех. Стресс тестирование – найти точку насыщения, когда нагрузка достигла критической, время отклика начинает расти хоть до этого не увеличивалась. Не ограничивайтесь только этими профилями, стройте зависимо от вашей системы
  • #36 - TPS could be improved by improving response time or by increasing concurrent users (кассира заставить работать быстрее, либо увеличить количество рабочих кас)
  • #37 - TPS could be improved by improving response time or by increasing concurrent users (кассира заставить работать быстрее, либо увеличить количество рабочих кас)