Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to QA Fest 2015. Владимир Примаков. Процесс нагрузочного тестирования и его планирование
Similar to QA Fest 2015. Владимир Примаков. Процесс нагрузочного тестирования и его планирование (20)
More from QAFest
More from QAFest (20)
Recently uploaded
Recently uploaded (20)
QA Fest 2015. Владимир Примаков. Процесс нагрузочного тестирования и его планирование
- 1. Performance and Load Testing Process. How to plan it www.qaexperts.pro
- 2. 22 12+ years general working experience in IT/QA (40+ projects) ---------- •QA Consultant / QA Manager at Ciklum Interactive •Load Testing Manager (3+ years, 15+ projects) •Automated Testing Manager •Co-owner at www.qaexperts.pro ---------- Skype: vladimir.primakov Linkedin: ua.linkedin.com/in/vladimirprimakov/ Email: v.v.primakov@gmail.com Some Words About Me Volodymyr Prymakov (Vladimir Primakov) www.qaexperts.pro
- 3. 33 • Introduction - 5 min • Main Part - 30 min • Questions - 5 min Presentation’s Structure Presentation Plan www.qaexperts.pro
- 4. 44 Assumptions We assume that: • Load & Performance testing tools are already chosen and used. • Load testing framework and infrastructure is build and fully functional. • Load testing team is already formed www.qaexperts.pro
- 5. 55 Load Testing Process 1. Planning 2. Evaluating Technical Risks 3. Prerequisites solving 4. Script implementation & adjustment 5. Test execution, analysis, and reporting 6. Summary reporting and conclusions www.qaexperts.pro
- 8. 88 Product Idea: Business domain, main specialties PRODUCT AND SCOPE Solution Architecture. The underlying technology. Pointer to Weak areas. www.qaexperts.pro
- 9. 99 Product and product parts IN and OUT of scope. Priority PRODUCT AND SCOPE 3D-Party services IN and OUT of scope. Mockups. Impact? www.qaexperts.pro
- 10. 1010 PRODUCT AND SCOPE www.qaexperts.pro 60% 30% 2%8% CLIENT PLATFORMS?
- 11. 1111 www.qaexperts.pro PRODUCT AND SCOPE COMMUNICATION TECHNOLOGY • Communication model and implementation for a) client <-> server side b) server-side <-> server-side • Protocols/exchange formats used in the communication • If there is any repetitive calls available in the communication? HTTP(S) (85%) Web-sockets (10%) TCP/IP (5%) TLS HTML Strings Binary XML JSON
- 12. 1212 PRODUCT AND SCOPE USAGE FLOWS / USER ACTIONS www.qaexperts.pro
- 13. 1313 PRODUCT AND SCOPE USAGE FLOWS / USER ACTIONS www.qaexperts.pro •Types of flows • Logged in and anonymous • Main Operations • Navigational •Approach (Stability and Traceability): •Try to isolate testable operation in a single flow •Make flows as shorter as you can •Avoid dependencies between flows
- 14. 1414 PRODUCT AND SCOPE USAGE FLOWS / USER ACTIONS www.qaexperts.pro Flow Type Flow % Anonymous Registration 3% Main Pages Navigation 10% Categories Navigation 15% Search and Navigate Item 20% Search and Buy item 15% Search and Add to Cart / Remove from cart 15% Compare Items 5% Logged in Add comment to / Ask question about an item 5% Add item to my wish list 5,00% Navigate to wish list 5,00% Edit Profile 2% Etc… … SUM 100,00% Registration Flow 1. Open main page 2. Go to the registration page 3. Fill in all the required data and click Register 4. Get the corresponding registration activation email - follow the registration activation link and login Example:
- 15. 1515 TEST ENVIRONMENTS www.qaexperts.pro Test environments for: 1. Evaluating the product / Technical Risk 2. Script implementation 3. Load test execution Difference between environment for test execution and target production environment? The corresponding risks. Possibility to minimize the difference.
- 16. 1616 LOAD PROFILES TYPES OF LOAD TESTS www.qaexperts.pro •10-100% Usual Load •Stress Load •Capacity Testing •Spike Stress Load •Soak Testing (24h-7d) •Volume Testing Load profiles Types of load tests
- 17. 1717 EXPECTED LOAD PASS-FAIL CRITERIA www.qaexperts.pro •Client-side criteria •Server-side criteria •Test Data
- 18. 1818 EXPECTED LOAD PASS-FAIL CRITERIA www.qaexperts.pro Client Which is actually: a) Number of registered users or b) Number of users per day or d) Correct number of concurrent users Response time for a operation or a request? Number of concurrent users =100000 Response time: 2 Sec
- 19. 1919 EXPECTED LOAD PASS-FAIL CRITERIA www.qaexperts.pro CLIENT-SIDE CRITERIA •Max number of concurrent users (anonymous and logged-in, active and passive) •% of Mobile/web/desktop/terminal clients •Max rate of main actions/operations per min, e.g.: 100 logins/min, 10 registration/min •Max response times for key pages and operations, e.g.: login – 3 sec, registration – 5 sec •Acceptable % of fails, e.g.: 2% Availability of such statistics from web-logs, Google analytics, and other tools/places?
- 20. 2020 EXPECTED LOAD PASS-FAIL CRITERIA www.qaexperts.pro SERVER-SIDE CRITERIA •Max Resource Consumption: CPU (e.g. 80%), RAM (e.g. 80%), Disk in/out, Network, etc. These parameters should be monitored within every node of a certain component •Max Errors Rate (in logs, etc.) e.g. 0.5%
- 21. 2121 EXPECTED LOAD PASS-FAIL CRITERIA www.qaexperts.pro TEST DATA •Users •User’s Data •Product’s Data --------------------------- 1. Type of Data, 2. Size and Amount of Data
- 22. 2222 OTHER NUANCES www.qaexperts.pro •Scheduling services? •Client’s internet connection speed? •Static content hosting: CDN or own servers? •Target region (USA, Europe, etc.)?
- 23. 2323 ORGANIZATIONAL MOMENTS www.qaexperts.pro •Project Milestones? •Date and Time frames for load testing runs? •Stakeholders? •Organizational, Business, Technical, and Server-side support contacts
- 24. 2424 • Difference between Target and Test environments • Risk of using service- mockups • Risk of negatively impacting the production environment or 3d party services • Wrong target load expectations • … www.qaexperts.pro TYPICAL RISKS
- 25. 2525 • Product & Solution successfully passed UAT • All prerequisites are solved • No unexpected deploys • … www.qaexperts.pro DEPENDENCIES / ASSUMPTIONS
- 28. 2828 www.qaexperts.pro PREREQUISITES SOLVING PLANNING IS DONE! Non-functional requirements and user flows are defined.
- 29. 2929 www.qaexperts.pro PREREQUISITES SOLVING PREPARE TEST ENVIRONMENT! • Test environments availability and access. Whitelisting load injectors. • Test environment and product stability. • Sever-side monitoring tools. • Test environment alignment to the production environment. • Test users and test data.
- 31. 3131 www.qaexperts.pro SCRIPT IMPLEMENTATION 1. Type of scripts: • Logged-in/Anonymous • Main flows scripts, • Navigational Scripts, • Repetitive calls scripts 2. Adjust scripts to the target environments 3. Regularly Update scripts to product changes 4. Define and follow script readiness criteria
- 33. 3333 www.qaexperts.pro PLAN TEST EXECUTION a) Plan load gradually (to eliminate Risks): 10% 25% 50% 75% 100% b) Or start load from Capacity test with long ramp-up period (to get fast results) Define load schedule: Types of load tests, dates, and timeframes for test runs
- 34. 3434 www.qaexperts.pro • Type of test/Goal • Date & Time of test • Server sider conditions and configuration; product version. • % of load • Ramp-up, Sustain, Ramp-down • Scripts (Flows) to run • Number of concurrent users in general and per script • Expected operations rate • Responsible specialists and their contacts in case of emergency. PLAN A RUN PLAN TEST EXECUTION
- 35. 3535 www.qaexperts.pro TEST EXECUTION • Make sure the server side support’s specialists are available • Do smoke test • Run Test: • Monitor Test product health: • Client • Response time growth • Error rate growth • Error types • Server • Resource consumptions • Errors • Servers availability • Communicate the results in real time • Be ready to Stop Test in case of Emergency! • Monitor Load Injectors health • If possible, play manually with the product under load
- 36. 3636 www.qaexperts.pro TEST RESULTS ANALYSIS AND REPORTING Client Side: • Response time-based capacity point • Errors-based capacity point • Failed scripts and transactions • Slow scripts and slow transactions • Errors details: types and amount • Max transactions/operations rates reached. • Timely trends and correlation with concurrent users number • Main bottlenecks concussions Server-side: • Resources consumptions • Errors and log-messages • Slow DB transactions • Correlation with client side metrics
- 37. 3737 www.qaexperts.pro Execute Load Test Meet Pass Criteria? Do Profiling Load Test Fix Solution/ Product TEST EXECUTION PROCESS Ready For Release YES NO If possible, the final step of load testing is testing on the prod environment under conditions similar to the real ones. Update Scripts Analyze Results
- 39. 3939 www.qaexperts.pro SUMMARY REPORTING AND CONCLUSIONS 1. Compare test runs: types of tests, server-side conditions, client and servers-side metrics 2. Conclusion about readiness to release. 3. Recommendations on further steps: a. Next load tests required. b. How to survive the expected load in case of not meeting the PASS criteria.
- 40. 4141 GENERAL SCHEDULE & ESTIMATION Weeks 1 2 3 4 5 6 7 8 9 Planning Evaluating Technical Risks Prerequisites solving Script implementation & adjustment Test execution, analysis, and reporting Summary reporting and conclusions 80% 50% EFFORTS AND DURATION www.qaexperts.pro
- 41. 4242 LOAD TESTING IN SDLS • In ongoing development – Verifying and validating component, queue of components, and integration related performance & robustness. • Before Release – Verifying and validating the whole product performance & robustness before release. • Maintenance – Verify and validating architectural, configurational, capacity-related, db-related, and integration-related changes. WHEN TO PERFORM www.qaexperts.pro