Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Continuous performance testing


Published on

Доклад Михаила Клокова на SQA Days-22. 17-18 ноября 2017. Санкт-Петербург, Россия

Published in: Education
  • Be the first to comment

Continuous performance testing

  1. 1. Continuous Performance TestingMikhail Klokov, QA Technical Director 17/11/17
  2. 2. 2 Agenda  Section 1: Introduction to CPT  Section 2: CPT solution  Section 2: Jagger CPT Solution  Section 4: Customer Success Story
  3. 3. 3 Why performance testing?  2012: Research showed that Amazon would lose $1.6 billion in sales every year if its site took one more second to load.  2013: 39% of e-retailers claimed they lost money last year due to performance or stability problems.  2014: The web performance monitoring company Catchpoint Systems looked at aggregate performance on Black Friday and compared it to the same timeframe in 2013.The results are notable: desktop web pages were 19.85 percent slower, while mobile web pages were a whopping 57.21 percent slower.  2015: Some major e-retailers’ sites buckled under the pressure of heavy holiday traffic during 2015’s Cyber Monday peak traffic times.
  4. 4. 4 Introduction to CPT
  5. 5. 5 Performance only as sign off Commit Build Unit Smoke Regression Integration Release deployment Performance tests and sign off Security sign off UAT sign off Functional E2E Release Candidate
  6. 6. 6 Performance impact of each change Add new feature Functionality check Performance impact
  7. 7. 7 Principles  Divide and conquer: stage your test execution  One throat to choke: target APIs for all testing goals  Keep your eyes on the prize: get your risks and KPIs right  Automate everything: one-click automation must be end-to-end  Analyze this: discover, visualize and deliver performance insights  Retain all results: value of data grows over time
  8. 8. 8 Include performance tests to CI loop Commit Build Unit Smoke Perf SmokeRegression Perf Regression Integration Perf E2E Release deployment Performance sign off Security sign off UAT sign off Functional E2E Release Candidate
  9. 9. 9 CPT goals  Certify that the application meets performance targets.  Plan capacity and manage growth.  Track useful performance metrics.  Identify load-related weaknesses and track SLA break points.
  10. 10. 10 CPT Solution
  11. 11. 11 Solution parts CPT PLATFORM  Create test loads for our SUT (System Under Test)  Collect metrics and store the results  Provide an analytics and visualization engine  Integrate with the CI/CD pipeline TEST DATA MANAGEMENT  Generate data based on existing subsets  Always use fresh test data  Compare test results on test data and production  Mask any sensitive test data ENVIRONMENT MANAGEMENT  Cloud  Provisioning on demand  Automated configuration  Automated deployment
  12. 12. 12 CPT framework requirement  Load  Generation different types of load (TPS, RPS, etc.)  Distributed load  Complex user scenarios for load  Visualization  Summary of test results  Detailed view during test run  Comparison of test runs and building trends  Metric  Standard performance metrics (response time, throughput, etc)  System and JVM metrics (CPU, memory, disk, network, etc)  Custom metrics (specific for SUT)  Results of validators (functional verification of the SUT responses)  Integration with CI platforms
  13. 13. 13 Jagger Grid Dynamics’ CPT platform
  14. 14. 14 Bird’s Eye View on Jagger
  15. 15. 15 Automation Decision Making  Baseline build is established as a reference point  New build results are compared with baseline  Results analysis: Pass when value is in range (Lower Warning Threshold *ref .. Upper Warning Threshold*ref) Warning when value is in range (Lower Error Threshold*ref .. Lower Warning Threshold*ref) OR (Upper Warning Threshold*ref .. Upper Error Threshold*ref) Error when value is less than Lower Error Threshold*ref OR is greater than Upper Error Threshold*ref
  16. 16. 16 Summary view
  17. 17. 17 Trends between different test sessions
  18. 18. 18 Detailed report: Parameters during test execution
  19. 19. 19 Jagger specifications  General Performance test in Jagger – Java project To develop test, some basic knowledge in programming is required Simple tests can be implemented just by XML editing  How to start Download and start test project example: Jagger Getting Started
  20. 20. 20 Customer Success Story
  21. 21. 21 CPT: Search and Browse for large eCommerce site 21 Challenges for delivery from testing perspective Solution overview Continuous performance testing platform  Jagger based performance testing platform with load generation and full performance and JVM metrics collection, automated trend analysis.  Evaluated SLA and critical metrics for all part our SUT (System Under Test) and automatically decision making based on them  Jagger WebUI for analytics and visualization. Compare test and session results, build trend, monitoring parameters during test.  Integrate with the CI/CD pipeline Continuous delivery platform  Jenkins based multi-stage testing pipelines  Cloud based deployment and configuration of all artifacts  Daily deployment into UAT environments  Bi-weekly deployment to production  Search engine has very flexible configuration  Complex business requirements  Platform consists of many collaborating services, including NoSQL database, SQL database, clustered search engine, two clustered services  Strict Non-functional and performance requirement Results  Reduce time for investigation of performance issues from weeks to 1-2 days  No critical or major performance defects in production in 3 years  1-2 performance issues/year discovered in pre-release performance certification
  22. 22. 22 Thank you! Mikhail Klokov: Grid Dynamics blog: Follow up on twitter: @griddynamics Jagger: We are hiring!
  23. 23. Privileged and confidential About Grid Dynamics Founded in 2006, Grid Dynamics is an engineering services company built on the premise that cloud computing is disruptive within the enterprise technology landscape. Since that time, we’ve had the privilege to help companies like Microsoft, eBay, PayPal, Cisco, Macy’s, Yahoo, ING, Bank of America, Kohl's, among others, to re-architect their core mission-critical systems, develop new cloud services, accelerate innovation cycles, increase software quality, and automate application management. Grid Dynamics has multiple locations in the USA and Europe, and employs over 700 expert engineers worldwide.