Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Performance and Load
Testing Process.
How to plan it
www.qaexperts.pro
22
12+ years general working experience in IT/QA (40+ projects)
----------
•QA Consultant / QA Manager at Ciklum Interacti...
33
• Introduction - 5 min
• Main Part - 30 min
• Questions - 5 min
Presentation’s Structure
Presentation Plan
www.qaexpert...
44
Assumptions
We assume that:
• Load & Performance testing tools are already chosen and used.
• Load testing framework an...
55
Load Testing Process
1. Planning
2. Evaluating Technical Risks
3. Prerequisites solving
4. Script implementation & adju...
66
1. PLANNING
www.qaexperts.pro
77
PRODUCT AND SCOPE
www.qaexperts.pro
88
Product Idea:
Business domain, main specialties
PRODUCT AND SCOPE
Solution Architecture.
The underlying technology.
Poi...
99
Product and product parts
IN and OUT of scope.
Priority
PRODUCT AND SCOPE
3D-Party services
IN and OUT of scope.
Mockup...
1010
PRODUCT AND SCOPE
www.qaexperts.pro
60%
30%
2%8%
CLIENT PLATFORMS?
1111 www.qaexperts.pro
PRODUCT AND SCOPE
COMMUNICATION TECHNOLOGY
• Communication model and implementation for
a) client <...
1212
PRODUCT AND SCOPE
USAGE FLOWS / USER ACTIONS
www.qaexperts.pro
1313
PRODUCT AND SCOPE
USAGE FLOWS / USER ACTIONS
www.qaexperts.pro
•Types of flows
• Logged in and anonymous
• Main Opera...
1414
PRODUCT AND SCOPE
USAGE FLOWS / USER ACTIONS
www.qaexperts.pro
Flow Type Flow %
Anonymous
Registration 3%
Main Pages ...
1515
TEST ENVIRONMENTS
www.qaexperts.pro
Test environments for:
1. Evaluating the product / Technical Risk
2. Script imple...
1616
LOAD PROFILES
TYPES OF LOAD TESTS
www.qaexperts.pro
•10-100% Usual Load
•Stress Load
•Capacity Testing
•Spike Stress ...
1717
EXPECTED LOAD
PASS-FAIL CRITERIA
www.qaexperts.pro
•Client-side criteria
•Server-side criteria
•Test Data
1818
EXPECTED LOAD
PASS-FAIL CRITERIA
www.qaexperts.pro
Client
Which is actually:
a) Number of registered users or
b) Numb...
1919
EXPECTED LOAD
PASS-FAIL CRITERIA
www.qaexperts.pro
CLIENT-SIDE CRITERIA
•Max number of concurrent users (anonymous an...
2020
EXPECTED LOAD
PASS-FAIL CRITERIA
www.qaexperts.pro
SERVER-SIDE CRITERIA
•Max Resource Consumption:
CPU (e.g. 80%), RA...
2121
EXPECTED LOAD
PASS-FAIL CRITERIA
www.qaexperts.pro
TEST DATA
•Users
•User’s Data
•Product’s Data
--------------------...
2222
OTHER NUANCES
www.qaexperts.pro
•Scheduling services?
•Client’s internet
connection speed?
•Static content hosting:
C...
2323
ORGANIZATIONAL MOMENTS
www.qaexperts.pro
•Project Milestones?
•Date and Time frames for
load testing runs?
•Stakehold...
2424
• Difference between Target
and Test environments
• Risk of using service-
mockups
• Risk of negatively impacting
the...
2525
• Product & Solution
successfully passed UAT
• All prerequisites are solved
• No unexpected deploys
• …
www.qaexperts...
2626
2. EVALUATING
TECHNICAL
RISKS
www.qaexperts.pro
2727
3.
PREREQUISITES
SOLVING
www.qaexperts.pro
2828 www.qaexperts.pro
PREREQUISITES SOLVING
PLANNING IS DONE!
Non-functional requirements
and user flows are defined.
2929 www.qaexperts.pro
PREREQUISITES SOLVING
PREPARE TEST ENVIRONMENT!
• Test environments availability
and access. Whitel...
3030
4.
SCRIPT
IMPLEMENTATION
www.qaexperts.pro
3131 www.qaexperts.pro
SCRIPT IMPLEMENTATION
1. Type of scripts:
• Logged-in/Anonymous
• Main flows scripts,
• Navigationa...
3232
5.
TEST EXECUTION,
ANALYSIS, AND
REPORTING
www.qaexperts.pro
3333
www.qaexperts.pro
PLAN TEST EXECUTION
a) Plan load gradually (to eliminate Risks):
10%
25%
50%
75%
100%
b) Or start l...
3434 www.qaexperts.pro
• Type of test/Goal
• Date & Time of test
• Server sider conditions and configuration; product
vers...
3535 www.qaexperts.pro
TEST EXECUTION
• Make sure the server side support’s specialists are available
• Do smoke test
• Ru...
3636 www.qaexperts.pro
TEST RESULTS ANALYSIS
AND REPORTING
Client Side:
• Response time-based capacity point
• Errors-base...
3737 www.qaexperts.pro
Execute
Load Test
Meet
Pass
Criteria?
Do
Profiling
Load Test
Fix
Solution/
Product
TEST EXECUTION P...
3838
6.
SUMMARY
REPORTING AND
CONCLUSIONS
www.qaexperts.pro
3939
www.qaexperts.pro
SUMMARY REPORTING
AND CONCLUSIONS
1. Compare test runs: types of tests,
server-side conditions, cli...
4141
GENERAL SCHEDULE & ESTIMATION
Weeks 1 2 3 4 5 6 7 8 9
Planning
Evaluating Technical Risks
Prerequisites solving
Scrip...
4242
LOAD TESTING IN SDLS
• In ongoing development – Verifying and validating
component, queue of components, and integrat...
4343
Thank you! ;)
QUESTIONS?
Upcoming SlideShare
Loading in …5
×

QA Fest 2015. Владимир Примаков. Процесс нагрузочного тестирования и его планирование

704 views

Published on

В этом докладе я хочу поделиться с вами подходом по планированию, организации, и проведению нагрузочного тестирования, выработанным и систематизированным на основании опыта проведения сервисов по нагрузочному тестированию более чем для дюжины проектов и систем различного масштаба. Основной акцент будет сделан на ряд тонких и важных моментов/нюансов обязательных для проведения нагрузочного тестирования полноценным и адекватным образом

Published in: Education
  • Be the first to comment

QA Fest 2015. Владимир Примаков. Процесс нагрузочного тестирования и его планирование

  1. 1. Performance and Load Testing Process. How to plan it www.qaexperts.pro
  2. 2. 22 12+ years general working experience in IT/QA (40+ projects) ---------- •QA Consultant / QA Manager at Ciklum Interactive •Load Testing Manager (3+ years, 15+ projects) •Automated Testing Manager •Co-owner at www.qaexperts.pro ---------- Skype: vladimir.primakov Linkedin: ua.linkedin.com/in/vladimirprimakov/ Email: v.v.primakov@gmail.com Some Words About Me Volodymyr Prymakov (Vladimir Primakov) www.qaexperts.pro
  3. 3. 33 • Introduction - 5 min • Main Part - 30 min • Questions - 5 min Presentation’s Structure Presentation Plan www.qaexperts.pro
  4. 4. 44 Assumptions We assume that: • Load & Performance testing tools are already chosen and used. • Load testing framework and infrastructure is build and fully functional. • Load testing team is already formed www.qaexperts.pro
  5. 5. 55 Load Testing Process 1. Planning 2. Evaluating Technical Risks 3. Prerequisites solving 4. Script implementation & adjustment 5. Test execution, analysis, and reporting 6. Summary reporting and conclusions www.qaexperts.pro
  6. 6. 66 1. PLANNING www.qaexperts.pro
  7. 7. 77 PRODUCT AND SCOPE www.qaexperts.pro
  8. 8. 88 Product Idea: Business domain, main specialties PRODUCT AND SCOPE Solution Architecture. The underlying technology. Pointer to Weak areas. www.qaexperts.pro
  9. 9. 99 Product and product parts IN and OUT of scope. Priority PRODUCT AND SCOPE 3D-Party services IN and OUT of scope. Mockups. Impact? www.qaexperts.pro
  10. 10. 1010 PRODUCT AND SCOPE www.qaexperts.pro 60% 30% 2%8% CLIENT PLATFORMS?
  11. 11. 1111 www.qaexperts.pro PRODUCT AND SCOPE COMMUNICATION TECHNOLOGY • Communication model and implementation for a) client <-> server side b) server-side <-> server-side • Protocols/exchange formats used in the communication • If there is any repetitive calls available in the communication? HTTP(S) (85%) Web-sockets (10%) TCP/IP (5%) TLS HTML Strings Binary XML JSON
  12. 12. 1212 PRODUCT AND SCOPE USAGE FLOWS / USER ACTIONS www.qaexperts.pro
  13. 13. 1313 PRODUCT AND SCOPE USAGE FLOWS / USER ACTIONS www.qaexperts.pro •Types of flows • Logged in and anonymous • Main Operations • Navigational •Approach (Stability and Traceability): •Try to isolate testable operation in a single flow •Make flows as shorter as you can •Avoid dependencies between flows
  14. 14. 1414 PRODUCT AND SCOPE USAGE FLOWS / USER ACTIONS www.qaexperts.pro Flow Type Flow % Anonymous Registration 3% Main Pages Navigation 10% Categories Navigation 15% Search and Navigate Item 20% Search and Buy item 15% Search and Add to Cart / Remove from cart 15% Compare Items 5% Logged in Add comment to / Ask question about an item 5% Add item to my wish list 5,00% Navigate to wish list 5,00% Edit Profile 2% Etc… … SUM 100,00% Registration Flow 1. Open main page 2. Go to the registration page 3. Fill in all the required data and click Register 4. Get the corresponding registration activation email - follow the registration activation link and login Example:
  15. 15. 1515 TEST ENVIRONMENTS www.qaexperts.pro Test environments for: 1. Evaluating the product / Technical Risk 2. Script implementation 3. Load test execution Difference between environment for test execution and target production environment? The corresponding risks. Possibility to minimize the difference.
  16. 16. 1616 LOAD PROFILES TYPES OF LOAD TESTS www.qaexperts.pro •10-100% Usual Load •Stress Load •Capacity Testing •Spike Stress Load •Soak Testing (24h-7d) •Volume Testing Load profiles Types of load tests
  17. 17. 1717 EXPECTED LOAD PASS-FAIL CRITERIA www.qaexperts.pro •Client-side criteria •Server-side criteria •Test Data
  18. 18. 1818 EXPECTED LOAD PASS-FAIL CRITERIA www.qaexperts.pro Client Which is actually: a) Number of registered users or b) Number of users per day or d) Correct number of concurrent users  Response time for a operation or a request? Number of concurrent users =100000 Response time: 2 Sec
  19. 19. 1919 EXPECTED LOAD PASS-FAIL CRITERIA www.qaexperts.pro CLIENT-SIDE CRITERIA •Max number of concurrent users (anonymous and logged-in, active and passive) •% of Mobile/web/desktop/terminal clients •Max rate of main actions/operations per min, e.g.: 100 logins/min, 10 registration/min •Max response times for key pages and operations, e.g.: login – 3 sec, registration – 5 sec •Acceptable % of fails, e.g.: 2% Availability of such statistics from web-logs, Google analytics, and other tools/places?
  20. 20. 2020 EXPECTED LOAD PASS-FAIL CRITERIA www.qaexperts.pro SERVER-SIDE CRITERIA •Max Resource Consumption: CPU (e.g. 80%), RAM (e.g. 80%), Disk in/out, Network, etc. These parameters should be monitored within every node of a certain component •Max Errors Rate (in logs, etc.) e.g. 0.5%
  21. 21. 2121 EXPECTED LOAD PASS-FAIL CRITERIA www.qaexperts.pro TEST DATA •Users •User’s Data •Product’s Data --------------------------- 1. Type of Data, 2. Size and Amount of Data
  22. 22. 2222 OTHER NUANCES www.qaexperts.pro •Scheduling services? •Client’s internet connection speed? •Static content hosting: CDN or own servers? •Target region (USA, Europe, etc.)?
  23. 23. 2323 ORGANIZATIONAL MOMENTS www.qaexperts.pro •Project Milestones? •Date and Time frames for load testing runs? •Stakeholders? •Organizational, Business, Technical, and Server-side support contacts
  24. 24. 2424 • Difference between Target and Test environments • Risk of using service- mockups • Risk of negatively impacting the production environment or 3d party services • Wrong target load expectations • … www.qaexperts.pro TYPICAL RISKS
  25. 25. 2525 • Product & Solution successfully passed UAT • All prerequisites are solved • No unexpected deploys • … www.qaexperts.pro DEPENDENCIES / ASSUMPTIONS
  26. 26. 2626 2. EVALUATING TECHNICAL RISKS www.qaexperts.pro
  27. 27. 2727 3. PREREQUISITES SOLVING www.qaexperts.pro
  28. 28. 2828 www.qaexperts.pro PREREQUISITES SOLVING PLANNING IS DONE! Non-functional requirements and user flows are defined.
  29. 29. 2929 www.qaexperts.pro PREREQUISITES SOLVING PREPARE TEST ENVIRONMENT! • Test environments availability and access. Whitelisting load injectors. • Test environment and product stability. • Sever-side monitoring tools. • Test environment alignment to the production environment. • Test users and test data.
  30. 30. 3030 4. SCRIPT IMPLEMENTATION www.qaexperts.pro
  31. 31. 3131 www.qaexperts.pro SCRIPT IMPLEMENTATION 1. Type of scripts: • Logged-in/Anonymous • Main flows scripts, • Navigational Scripts, • Repetitive calls scripts 2. Adjust scripts to the target environments 3. Regularly Update scripts to product changes 4. Define and follow script readiness criteria
  32. 32. 3232 5. TEST EXECUTION, ANALYSIS, AND REPORTING www.qaexperts.pro
  33. 33. 3333 www.qaexperts.pro PLAN TEST EXECUTION a) Plan load gradually (to eliminate Risks): 10% 25% 50% 75% 100% b) Or start load from Capacity test with long ramp-up period (to get fast results) Define load schedule: Types of load tests, dates, and timeframes for test runs
  34. 34. 3434 www.qaexperts.pro • Type of test/Goal • Date & Time of test • Server sider conditions and configuration; product version. • % of load • Ramp-up, Sustain, Ramp-down • Scripts (Flows) to run • Number of concurrent users in general and per script • Expected operations rate • Responsible specialists and their contacts in case of emergency. PLAN A RUN PLAN TEST EXECUTION
  35. 35. 3535 www.qaexperts.pro TEST EXECUTION • Make sure the server side support’s specialists are available • Do smoke test • Run Test: • Monitor Test product health: • Client • Response time growth • Error rate growth • Error types • Server • Resource consumptions • Errors • Servers availability • Communicate the results in real time • Be ready to Stop Test in case of Emergency! • Monitor Load Injectors health • If possible, play manually with the product under load
  36. 36. 3636 www.qaexperts.pro TEST RESULTS ANALYSIS AND REPORTING Client Side: • Response time-based capacity point • Errors-based capacity point • Failed scripts and transactions • Slow scripts and slow transactions • Errors details: types and amount • Max transactions/operations rates reached. • Timely trends and correlation with concurrent users number • Main bottlenecks concussions Server-side: • Resources consumptions • Errors and log-messages • Slow DB transactions • Correlation with client side metrics
  37. 37. 3737 www.qaexperts.pro Execute Load Test Meet Pass Criteria? Do Profiling Load Test Fix Solution/ Product TEST EXECUTION PROCESS Ready For Release YES NO If possible, the final step of load testing is testing on the prod environment under conditions similar to the real ones. Update Scripts Analyze Results
  38. 38. 3838 6. SUMMARY REPORTING AND CONCLUSIONS www.qaexperts.pro
  39. 39. 3939 www.qaexperts.pro SUMMARY REPORTING AND CONCLUSIONS 1. Compare test runs: types of tests, server-side conditions, client and servers-side metrics 2. Conclusion about readiness to release. 3. Recommendations on further steps: a. Next load tests required. b. How to survive the expected load in case of not meeting the PASS criteria.
  40. 40. 4141 GENERAL SCHEDULE & ESTIMATION Weeks 1 2 3 4 5 6 7 8 9 Planning Evaluating Technical Risks Prerequisites solving Script implementation & adjustment Test execution, analysis, and reporting Summary reporting and conclusions 80% 50% EFFORTS AND DURATION www.qaexperts.pro
  41. 41. 4242 LOAD TESTING IN SDLS • In ongoing development – Verifying and validating component, queue of components, and integration related performance & robustness. • Before Release – Verifying and validating the whole product performance & robustness before release. • Maintenance – Verify and validating architectural, configurational, capacity-related, db-related, and integration-related changes. WHEN TO PERFORM www.qaexperts.pro
  42. 42. 4343 Thank you! ;) QUESTIONS?

×