Download as PDF, PPTX
Uploaded byGrid Dynamics
Continuous Performance Testing
The document outlines the importance and methodology of Continuous Performance Testing (CPT), highlighting its necessity for maintaining optimal performance in e-commerce platforms. It provides a detailed overview of the CPT solutions offered by Grid Dynamics, utilizing the Jagger platform for performance testing and analysis, and showcases a customer success story that demonstrates significant improvements in performance management. Key goals of CPT include certifying application performance, planning capacity, and tracking performance metrics, leading to reduced investigation times for performance issues.
More Related Content
Similar to Continuous Performance Testing
Continuous Performance Testing
- 1. Continuous Performance Testing MikhailKlokov, QA Technical Director 17/11/17
- 2. 2 Agenda § Section 1:Introduction to CPT § Section 2: CPT solution § Section 2: Jagger CPT Solution § Section 4: Customer Success Story
- 3. 3 Why performance testing? §2012: Research showed that Amazon would lose $1.6 billion in sales every year if its site took one more second to load. § 2013: 39% of e-retailers claimed they lost money last year due to performance or stability problems. § 2014: The web performance monitoring company Catchpoint Systems looked at aggregate performance on Black Friday and compared it to the same timeframe in 2013.The results are notable: desktop web pages were 19.85 percent slower, while mobile web pages were a whopping 57.21 percent slower. § 2015: Some major e-retailers’ sites buckled under the pressure of heavy holiday traffic during 2015’s Cyber Monday peak traffic times.
- 4.
- 5. 5 Performance only assign off Commit Build Unit Smoke Regression Integration Release deployment Performance tests and sign off Security sign off UAT sign off Functional E2E Release Candidate
- 6. 6 Performance impact ofeach change Add new feature Functionality check Performance impact
- 7. 7 Include performance teststo CI loop Commit Build Unit Smoke Perf SmokeRegression Perf Regression Integration Perf E2E Release deployment Performance sign off Security sign off UAT sign off Functional E2E Release Candidate
- 8. 8 Principles § Divide andconquer: stage your test execution § One throat to choke: target APIs for all testing goals § Keep your eyes on the prize: get your risks and KPIs right § Automate everything: one-click automation must be end-to-end § Analyze this: discover, visualize and deliver performance insights § Retain all results: value of data grows over time
- 9. 9 CPT goals § Certifythat the application meets performance targets. § Plan capacity and manage growth. § Track useful performance metrics. § Identify load-related weaknesses and track SLA break points.
- 10.
- 11. 11 Solution parts § CPTPLATFORM – test framework, results storage, metrics collection, and visualization § TEST DATA MANAGEMENT – data manipulation to establish the CPT process § ENVIRONMENT MANAGEMENT – controlling the performance test environment life cycle
- 12. 12 Solution parts CPT PLATFORM §Create test loads for our SUT (System Under Test) § Collect metrics and store the results § Provide an analytics and visualization engine § Integrate with the CI/CD pipeline TEST DATA MANAGEMENT § Generate data based on existing subsets § Always use fresh test data § Compare test results on test data and production § Mask any sensitive test data ENVIRONMENT MANAGEMENT § Cloud § Provisioning on demand § Automated configuration § Automated deployment
- 13. 13 CPT framework requirement §Load § Generation different types of load (TPS, RPS, etc.) § Distributed load § Complex user scenarios for load § Visualization § Summary of test results § Detailed view during test run § Comparison of test runs and building trends § Metric § Standard performance metrics (response time, throughput, etc) § System and JVM metrics (CPU, memory, disk, network, etc) § Custom metrics (specific for SUT) § Results of validators (functional verification of the SUT responses) § Integration with CI platforms
- 14. 14 Jagger Grid Dynamics’ CPTplatform
- 15. 15 Bird’s Eye Viewon Jagger
- 16. 16 Jenkins Job StatusResult § Baseline build is established as a reference point § New build results are compared with baseline § Results analysis: Pass when value is in range (Lower Warning Threshold *ref .. Upper Warning Threshold*ref) Warning when value is in range (Lower Error Threshold*ref .. Lower Warning Threshold*ref) OR (Upper Warning Threshold*ref .. Upper Error Threshold*ref) Error when value is less than Lower Error Threshold*ref OR is greater than Upper Error Threshold*ref
- 17.
- 18. 18 Trends between differenttest sessions
- 19. 19 Detailed report: Parametersduring test execution
- 20. 20 Jagger specifications § General Performancetest in Jagger – Java project To develop test, some basic knowledge in programming is required Simple tests can be implemented just by XML editing § How to start Download and start test project example: Jagger Getting Started
- 21.
- 22. 22 CPT: Search andBrowse for large eCommerce site 22 Challenges for delivery from testing perspective Solution overview Continuous performance testing platform § Jagger based performance testing platform with load generation and full performance and JVM metrics collection, automated trend analysis. § Evaluated SLA and critical metrics for all part our SUT (System Under Test) and automatically decision making based on them § Jagger WebUI for analytics and visualization. Compare test and session results, build trend, monitoring parameters during test. § Integrate with the CI/CD pipeline Continuous delivery platform § Jenkins based multi-stage testing pipelines § Cloud based deployment and configuration of all artifacts § Daily deployment into UAT environments § Bi-weekly deployment to production § Search engine has very flexible configuration § Complex business requirements § Platform consists of many collaborating services, including NoSQL database, SQL database, clustered search engine, two clustered services § Strict Non-functional and performance requirement Results § Reduce time for investigation of performance issues from weeks to 1-2 days § No critical or major performance defects in production in 3 years § 1-2 performance issues/year discovered in pre-release performance certification
- 23. 23 Thank you! Mikhail Klokov:mklokov@griddynamics.com Grid Dynamics blog: blog.griddynamics.com Follow up on twitter: @griddynamics Jagger: jagger.griddynamics.net We are hiring! griddynamics.com/careers
- 24. Privileged and confidential www.griddynamics.com AboutGrid Dynamics Founded in 2006, Grid Dynamics is an engineering services company built on the premise that cloud computing is disruptive within the enterprise technology landscape. Since that time, we’ve had the privilege to help companies like Microsoft, eBay, PayPal, Cisco, Macy’s, Yahoo, ING, Bank of America, Kohl's, among others, to re- architect their core mission-critical systems, develop new cloud services, accelerate innovation cycles, increase software quality, and automate application management. Grid Dynamics has multiple locations in the USA and Europe, and employs over 700 expert engineers worldwide.