This document discusses Jagger, a continuous performance testing tool for mission-critical applications. It describes how Jagger is used to develop backend services for top retailers, load test clusters of 40+ nodes handling up to 50k transactions per second. The document outlines antipatterns in traditional performance testing and introduces Jagger's technical architecture and automated testing ecosystem. It provides lessons learned around automating testing, integrating monitoring data, profiling systems under test, validating responses, and managing large amounts of test results data.

1. Scalable eCommerce Platform Solutions
Jagger: Continuous Performance
Testing of Mission-Critical Applications
Ilya Katsov, Grid Dynamics

2. Scalable eCommerce Platform Solutions
4/29/2013 2
Performance
Testing
Tool
System
Under
Test
Report
What’s wrong?

3. Scalable eCommerce Platform Solutions
4/29/2013 3
What we do
• Develop backend services for 3 of Top-30 US Retailers
• Production clusters of 40 nodes and more
• Load up to 50K TPS
• Heavy computations
in navigational queries:

4. Scalable eCommerce Platform Solutions
4/29/2013 4
Antipatterns of Performance Testing
• Acceptance performance testing
• Independent functional and performance testing
• Poor tracking of testing results
• Insufficient introspection of the system under test
• Failover is tested in production

5. Scalable eCommerce Platform Solutions
4/29/2013 5
Performance
Testing
Tool
System
Under
Test
Report
What’s wrong?

6. Scalable eCommerce Platform Solutions
4/29/2013 6
Performance Testing Ecosystem
Development
Environment
Performance
Environment
QA
Environment
Production
Environment
Functional
Tests
Profiling Performance
Tests
Monitoring
Release
Manager
Performance
Test Lead
Performance
Architect
Technical
Lead

7. Scalable eCommerce Platform Solutions
4/29/2013 7
Slave Slave Slave SlaveSlave
Master
SuT
SuT
Agent
SuT
SuT
Agent
SuT
SuT
Agent
SuT
SuT
Agent
SuT
SuT
Agent
SuT
SuT
Agent
SuT
SuT
Agent
Jagger Technical Architecture
Web UI
HDFS
MySQL
Zookeeper

8. Scalable eCommerce Platform Solutions
4/29/2013 8
Lessons learned: Automate. Automate to scale.
• Completely automated flows
• Jenkins plugin for automatic deployment
• Results warehousing in the database

9. Scalable eCommerce Platform Solutions
4/29/2013 9
Lessons learned: Automate. Automate to scale.
• Decision makers that can access all collected data
• Session comparison
• VCS-friendly configuration in plain text files

10. Scalable eCommerce Platform Solutions
4/29/2013 10
Lessons learned: Monitoring is everything.
• OS-level and JVM-level metrics
• Ability to load external metrics
• All metrics can be involved into automatic decision
making
• Jagger can be used to monitor an external process:
Activity
Jagger
Agent
Jagger Cluster
CI Server

11. Scalable eCommerce Platform Solutions
4/29/2013 11
Lessons learned: Profile system under test
• Performance issues investigation – a typical flow:
Problem ResolvedFix
Performance
Testing
Performance
Problem Detected

12. Scalable eCommerce Platform Solutions
4/29/2013 12
Lessons learned: Profile system under test
• Sampling profiler for JVM (stacktraces via JMX)
• Profiler contains intelligence to detect hot spots
Performance
Problem Detected
Problem ResolvedFix
Profiling
results
Performance
Testing

13. Scalable eCommerce Platform Solutions
4/29/2013 13
Lessons learned: Validate responses
• Custom validators
• Record results of single-user test and use for validation

14. Scalable eCommerce Platform Solutions
4/29/2013 14
Lessons learned: Manage your data well.
• Statistics from one of our accounts:
• About 100 test cases
• 20 scheduled test sessions weekly + ad hoc testing
• 30GB of test results, more than 1500 retained sessions

15. Scalable eCommerce Platform Solutions
4/29/2013 15
Lessons learned: Manage your data well.
• Reports - PDF and web UI

16. Scalable eCommerce Platform Solutions
4/29/2013 16
Results and Roadmap
Successful:
• Continuous testing
• Monitoring
• Validation
• Data management
Not very successful:
• Continuous robustness
testing
• Integration with production
monitoring

17. Scalable eCommerce Platform Solutions
4/29/2013 17
jagger@griddynamics.com
Contact Us
https://jagger.griddynamics.net
Distribution and Documentation