1. Continuous Performance
TestingMikhail Klokov, QA Technical Director
17/11/17

2. 2
Agenda
 Section 1: Introduction to CPT
 Section 2: CPT solution
 Section 2: Jagger CPT Solution
 Section 4: Customer Success Story

3. 3
Why performance testing?
 2012: Research showed that Amazon would lose $1.6 billion in sales every year if
its site took one more second to load.
 2013: 39% of e-retailers claimed they lost money last year due to performance or
stability problems.
 2014: The web performance monitoring company Catchpoint Systems looked at
aggregate performance on Black Friday and compared it to the same timeframe in
2013.The results are notable: desktop web pages were 19.85 percent slower, while
mobile web pages were a whopping 57.21 percent slower.
 2015: Some major e-retailers’ sites buckled under the pressure of heavy holiday
traffic during 2015’s Cyber Monday peak traffic times.

4. 4
Introduction to CPT

5. 5
Performance only as sign off
Commit
Build
Unit
Smoke
Regression
Integration
Release deployment
Performance tests and sign off
Security sign off
UAT sign off
Functional E2E
Release Candidate

6. 6
Performance impact of each change
Add new
feature
Functionality
check
Performance
impact

7. 7
Principles
 Divide and conquer: stage your test execution
 One throat to choke: target APIs for all testing goals
 Keep your eyes on the prize: get your risks and KPIs right
 Automate everything: one-click automation must be end-to-end
 Analyze this: discover, visualize and deliver performance insights
 Retain all results: value of data grows over time

8. 8
Include performance tests to CI loop
Commit
Build
Unit
Smoke
Perf SmokeRegression
Perf
Regression
Integration
Perf E2E
Release deployment
Performance sign off
Security sign off
UAT sign off
Functional E2E
Release Candidate

9. 9
CPT goals
 Certify that the application meets performance targets.
 Plan capacity and manage growth.
 Track useful performance metrics.
 Identify load-related weaknesses and track SLA break points.

10. 10
CPT Solution

11. 11
Solution parts
CPT PLATFORM
 Create test loads for our SUT (System Under Test)
 Collect metrics and store the results
 Provide an analytics and visualization engine
 Integrate with the CI/CD pipeline
TEST DATA MANAGEMENT
 Generate data based on existing subsets
 Always use fresh test data
 Compare test results on test data and production
 Mask any sensitive test data
ENVIRONMENT MANAGEMENT
 Cloud
 Provisioning on demand
 Automated configuration
 Automated deployment

12. 12
CPT framework requirement
 Load
 Generation different types of load (TPS,
RPS, etc.)
 Distributed load
 Complex user scenarios for load
 Visualization
 Summary of test results
 Detailed view during test run
 Comparison of test runs and building
trends
 Metric
 Standard performance metrics (response
time, throughput, etc)
 System and JVM metrics (CPU, memory,
disk, network, etc)
 Custom metrics (specific for SUT)
 Results of validators (functional
verification of the SUT responses)
 Integration with CI platforms

13. 13
Jagger
Grid Dynamics’ CPT platform

14. 14
Bird’s Eye View on Jagger

15. 15
Automation Decision Making
 Baseline build is established as a reference point
 New build results are compared with baseline
 Results analysis:
Pass when value is in range (Lower Warning Threshold *ref .. Upper Warning Threshold*ref)
Warning when value is in range (Lower Error Threshold*ref .. Lower Warning Threshold*ref) OR
(Upper Warning Threshold*ref .. Upper Error
Threshold*ref)
Error when value is less than Lower Error Threshold*ref OR is greater than Upper Error Threshold*ref

16. 16
Summary view

17. 17
Trends between different test sessions

18. 18
Detailed report: Parameters during test execution

19. 19
Jagger specifications
 General
Performance test in Jagger – Java project
To develop test, some basic knowledge in programming is required
Simple tests can be implemented just by XML editing
 How to start
Download and start test project example: Jagger Getting Started

20. 20
Customer Success Story

21. 21
CPT: Search and Browse for large eCommerce site
21
Challenges for delivery from
testing perspective Solution overview
Continuous performance testing platform
 Jagger based performance testing platform with load
generation and full performance and JVM metrics
collection, automated trend analysis.
 Evaluated SLA and critical metrics for all part our SUT
(System Under Test) and automatically decision making
based on them
 Jagger WebUI for analytics and visualization. Compare
test and session results, build trend, monitoring parameters
during test.
 Integrate with the CI/CD pipeline
Continuous delivery platform
 Jenkins based multi-stage testing pipelines
 Cloud based deployment and configuration of all artifacts
 Daily deployment into UAT environments
 Bi-weekly deployment to production
 Search engine has very flexible
configuration
 Complex business requirements
 Platform consists of many collaborating
services, including NoSQL database, SQL
database, clustered search engine, two
clustered services
 Strict Non-functional and performance
requirement
Results
 Reduce time for investigation of
performance issues from weeks to 1-2
days
 No critical or major performance defects
in production in 3 years
 1-2 performance issues/year discovered
in pre-release performance certification

22. 22
Thank you!
Mikhail Klokov: mklokov@griddynamics.com
Grid Dynamics blog: blog.griddynamics.com
Follow up on twitter: @griddynamics
Jagger: jagger.griddynamics.net
We are hiring!
griddynamics.com/careers

23. Privileged and confidential
www.griddynamics.com
About Grid Dynamics
Founded in 2006, Grid Dynamics is an engineering services company built on
the premise that cloud computing is disruptive within the enterprise technology
landscape. Since that time, we’ve had the privilege to help companies like
Microsoft, eBay, PayPal, Cisco, Macy’s, Yahoo, ING, Bank of America, Kohl's,
among others, to re-architect their core mission-critical systems, develop new
cloud services, accelerate innovation cycles, increase software quality, and
automate application management.
Grid Dynamics has multiple locations in the USA and Europe, and employs over
700 expert engineers worldwide.