More Related Content


More from QAFest(20)


QA Fest 2015. Владимир Примаков. Процесс нагрузочного тестирования и его планирование

  1. Performance and Load Testing Process. How to plan it
  2. 22 12+ years general working experience in IT/QA (40+ projects) ---------- •QA Consultant / QA Manager at Ciklum Interactive •Load Testing Manager (3+ years, 15+ projects) •Automated Testing Manager •Co-owner at ---------- Skype: vladimir.primakov Linkedin: Email: Some Words About Me Volodymyr Prymakov (Vladimir Primakov)
  3. 33 • Introduction - 5 min • Main Part - 30 min • Questions - 5 min Presentation’s Structure Presentation Plan
  4. 44 Assumptions We assume that: • Load & Performance testing tools are already chosen and used. • Load testing framework and infrastructure is build and fully functional. • Load testing team is already formed
  5. 55 Load Testing Process 1. Planning 2. Evaluating Technical Risks 3. Prerequisites solving 4. Script implementation & adjustment 5. Test execution, analysis, and reporting 6. Summary reporting and conclusions
  6. 66 1. PLANNING
  8. 88 Product Idea: Business domain, main specialties PRODUCT AND SCOPE Solution Architecture. The underlying technology. Pointer to Weak areas.
  9. 99 Product and product parts IN and OUT of scope. Priority PRODUCT AND SCOPE 3D-Party services IN and OUT of scope. Mockups. Impact?
  11. 1111 PRODUCT AND SCOPE COMMUNICATION TECHNOLOGY • Communication model and implementation for a) client <-> server side b) server-side <-> server-side • Protocols/exchange formats used in the communication • If there is any repetitive calls available in the communication? HTTP(S) (85%) Web-sockets (10%) TCP/IP (5%) TLS HTML Strings Binary XML JSON
  13. 1313 PRODUCT AND SCOPE USAGE FLOWS / USER ACTIONS •Types of flows • Logged in and anonymous • Main Operations • Navigational •Approach (Stability and Traceability): •Try to isolate testable operation in a single flow •Make flows as shorter as you can •Avoid dependencies between flows
  14. 1414 PRODUCT AND SCOPE USAGE FLOWS / USER ACTIONS Flow Type Flow % Anonymous Registration 3% Main Pages Navigation 10% Categories Navigation 15% Search and Navigate Item 20% Search and Buy item 15% Search and Add to Cart / Remove from cart 15% Compare Items 5% Logged in Add comment to / Ask question about an item 5% Add item to my wish list 5,00% Navigate to wish list 5,00% Edit Profile 2% Etc… … SUM 100,00% Registration Flow 1. Open main page 2. Go to the registration page 3. Fill in all the required data and click Register 4. Get the corresponding registration activation email - follow the registration activation link and login Example:
  15. 1515 TEST ENVIRONMENTS Test environments for: 1. Evaluating the product / Technical Risk 2. Script implementation 3. Load test execution Difference between environment for test execution and target production environment? The corresponding risks. Possibility to minimize the difference.
  16. 1616 LOAD PROFILES TYPES OF LOAD TESTS •10-100% Usual Load •Stress Load •Capacity Testing •Spike Stress Load •Soak Testing (24h-7d) •Volume Testing Load profiles Types of load tests
  17. 1717 EXPECTED LOAD PASS-FAIL CRITERIA •Client-side criteria •Server-side criteria •Test Data
  18. 1818 EXPECTED LOAD PASS-FAIL CRITERIA Client Which is actually: a) Number of registered users or b) Number of users per day or d) Correct number of concurrent users  Response time for a operation or a request? Number of concurrent users =100000 Response time: 2 Sec
  19. 1919 EXPECTED LOAD PASS-FAIL CRITERIA CLIENT-SIDE CRITERIA •Max number of concurrent users (anonymous and logged-in, active and passive) •% of Mobile/web/desktop/terminal clients •Max rate of main actions/operations per min, e.g.: 100 logins/min, 10 registration/min •Max response times for key pages and operations, e.g.: login – 3 sec, registration – 5 sec •Acceptable % of fails, e.g.: 2% Availability of such statistics from web-logs, Google analytics, and other tools/places?
  20. 2020 EXPECTED LOAD PASS-FAIL CRITERIA SERVER-SIDE CRITERIA •Max Resource Consumption: CPU (e.g. 80%), RAM (e.g. 80%), Disk in/out, Network, etc. These parameters should be monitored within every node of a certain component •Max Errors Rate (in logs, etc.) e.g. 0.5%
  21. 2121 EXPECTED LOAD PASS-FAIL CRITERIA TEST DATA •Users •User’s Data •Product’s Data --------------------------- 1. Type of Data, 2. Size and Amount of Data
  22. 2222 OTHER NUANCES •Scheduling services? •Client’s internet connection speed? •Static content hosting: CDN or own servers? •Target region (USA, Europe, etc.)?
  23. 2323 ORGANIZATIONAL MOMENTS •Project Milestones? •Date and Time frames for load testing runs? •Stakeholders? •Organizational, Business, Technical, and Server-side support contacts
  24. 2424 • Difference between Target and Test environments • Risk of using service- mockups • Risk of negatively impacting the production environment or 3d party services • Wrong target load expectations • … TYPICAL RISKS
  25. 2525 • Product & Solution successfully passed UAT • All prerequisites are solved • No unexpected deploys • … DEPENDENCIES / ASSUMPTIONS
  28. 2828 PREREQUISITES SOLVING PLANNING IS DONE! Non-functional requirements and user flows are defined.
  29. 2929 PREREQUISITES SOLVING PREPARE TEST ENVIRONMENT! • Test environments availability and access. Whitelisting load injectors. • Test environment and product stability. • Sever-side monitoring tools. • Test environment alignment to the production environment. • Test users and test data.
  31. 3131 SCRIPT IMPLEMENTATION 1. Type of scripts: • Logged-in/Anonymous • Main flows scripts, • Navigational Scripts, • Repetitive calls scripts 2. Adjust scripts to the target environments 3. Regularly Update scripts to product changes 4. Define and follow script readiness criteria
  33. 3333 PLAN TEST EXECUTION a) Plan load gradually (to eliminate Risks): 10% 25% 50% 75% 100% b) Or start load from Capacity test with long ramp-up period (to get fast results) Define load schedule: Types of load tests, dates, and timeframes for test runs
  34. 3434 • Type of test/Goal • Date & Time of test • Server sider conditions and configuration; product version. • % of load • Ramp-up, Sustain, Ramp-down • Scripts (Flows) to run • Number of concurrent users in general and per script • Expected operations rate • Responsible specialists and their contacts in case of emergency. PLAN A RUN PLAN TEST EXECUTION
  35. 3535 TEST EXECUTION • Make sure the server side support’s specialists are available • Do smoke test • Run Test: • Monitor Test product health: • Client • Response time growth • Error rate growth • Error types • Server • Resource consumptions • Errors • Servers availability • Communicate the results in real time • Be ready to Stop Test in case of Emergency! • Monitor Load Injectors health • If possible, play manually with the product under load
  36. 3636 TEST RESULTS ANALYSIS AND REPORTING Client Side: • Response time-based capacity point • Errors-based capacity point • Failed scripts and transactions • Slow scripts and slow transactions • Errors details: types and amount • Max transactions/operations rates reached. • Timely trends and correlation with concurrent users number • Main bottlenecks concussions Server-side: • Resources consumptions • Errors and log-messages • Slow DB transactions • Correlation with client side metrics
  37. 3737 Execute Load Test Meet Pass Criteria? Do Profiling Load Test Fix Solution/ Product TEST EXECUTION PROCESS Ready For Release YES NO If possible, the final step of load testing is testing on the prod environment under conditions similar to the real ones. Update Scripts Analyze Results
  39. 3939 SUMMARY REPORTING AND CONCLUSIONS 1. Compare test runs: types of tests, server-side conditions, client and servers-side metrics 2. Conclusion about readiness to release. 3. Recommendations on further steps: a. Next load tests required. b. How to survive the expected load in case of not meeting the PASS criteria.
  40. 4141 GENERAL SCHEDULE & ESTIMATION Weeks 1 2 3 4 5 6 7 8 9 Planning Evaluating Technical Risks Prerequisites solving Script implementation & adjustment Test execution, analysis, and reporting Summary reporting and conclusions 80% 50% EFFORTS AND DURATION
  41. 4242 LOAD TESTING IN SDLS • In ongoing development – Verifying and validating component, queue of components, and integration related performance & robustness. • Before Release – Verifying and validating the whole product performance & robustness before release. • Maintenance – Verify and validating architectural, configurational, capacity-related, db-related, and integration-related changes. WHEN TO PERFORM
  42. 4343 Thank you! ;) QUESTIONS?