This document discusses performance testing challenges for an agile development team working on a performance critical Java application. It estimates that manually executing performance tests against 9 configurations would take 1+ man-months. To address this, it evaluates options like adding more performance engineers, limiting tests and configurations, or automating performance testing. It recommends automating testing for benefits like running tests continuously and allowing small teams to efficiently test performance. The case study details how the team automated testing using JMeter, built a process integrated with TeamCity, and upgraded infrastructure to support concurrent testing. Automation reduced the testing cycle from over 1 man-month to 4 days, allowing more time for analysis and new testing while finding 17 issues.

1. © Copyright GlobalLogic 2011 1
Connect. Collaborate. Innovate.
Performance testing in Agile development.
Automation challenges.

2. © Copyright GlobalLogic 2011 2
Connect. Collaborate. Innovate.
Performance critical application
Performance critical java enterprise
application
Operating
systems
Application servers Database
servers
9 priority-driven configurations
two performance engineers

3. © Copyright GlobalLogic 2011 3
Connect. Collaborate. Innovate.
Time estimates
5 core load testing scenarios
On average manual execution of a single test takes 4 hours
(setup environment, execute a test, basic results analysis)
Estimated time to complete load testing against 9 configurations:
9 configurations * 5 test scenarios * 4 hours = 180 hours
1+ man-months
two-week sprints?

4. © Copyright GlobalLogic 2011 4
Connect. Collaborate. Innovate.
Before any work is started
1. Identify Test Environment
2. Identify Performance
Acceptance Criteria
3. Plan and Design Tests
4. Implement Test Design
5. Configure Test Environment
7. Analyze Results, Report,
and Retest
Most time-consuming
activities
Solutions
6. Execute Tests
Core performance testing
activities
More performance engineers ?
Limit the number of tests and
configurations?
Automation ?

5. © Copyright GlobalLogic 2011 5
Connect. Collaborate. Innovate.
More performance engineers
Cons:
• Noticeably bigger budget of performance testing effort
• Additional activities to manage a team, usually with dedicated
lead/manager
• Staff trainings: methodology, tools
• Time consuming and boring tasks are performed by
performance engineers
• Test LAB investments
• Performance engineer positions are hard to fill
Pros:
• There is no timemoney investment in automation
• More work done in less time
• Can be used when automation is impossible due to product
complexity and architecture limitations
• Can be applicable to a short-term project

6. © Copyright GlobalLogic 2011 6
Connect. Collaborate. Innovate.
Limit the number of tests and configurations
Pros:
• There is no timemoney investment in automation
• Focus on most commonly used configurations
• There is no need to expand performance testing team
• Can be applicable to short-term project
• Can be used when risk to miss performance issue is
considered and acceptable
Cons:
• There is a high risk to miss environment specific performance
issue which could cause performance problems in production
• Very risky when an application is performance critical
• The approach doesn’t guarantee that a team will have enough
time to investigate problems deeper
• Time-consuming and boring tasks are performed by
performance engineers

7. © Copyright GlobalLogic 2011 7
Connect. Collaborate. Innovate.
Automation
Pros:
• Automated performance tests 24X7
• Continuous integration process
• Concurrent performance tests
• Early warning of performance bottleneck or performance
issue
• Performance testing with a small team
• More time for results analysis and investigations
• There is no need to expand a performance team
• Profitable for long running, performance critical projects
• Reusable by any person in a team
Cons:
• Timemoney investment to build automation
• Ongoing efforts to keep automation in a good state
• Usually not applicable for short-term projects
• Test LAB investments
• Commercial tools are expensive

8. © Copyright GlobalLogic 2011 8
Connect. Collaborate. Innovate.
Principles of performance testing in Agile
Make performance explicit
• Make it an explicit requirement upfront and throughout the process (non-functional
requirements can be dotted onto the corner of the story card for a functional
requirement)
Performance testing is a part of continuos integration process:
• Automated compilation
• Automated packaging
• Automated deployment
• Automated configuration
• Automated tests execution
• Automated analysis
• Automated result archiving - can check raw output if discover issues later
• Automated scheduling. Overnight runs.
Use concurrent performance testing
Iterate and increment
Get feedback

9. © Copyright GlobalLogic 2011 9
Connect. Collaborate. Innovate.
Ant, Jmeter, Scalable Vector Graphics case study

10. © Copyright GlobalLogic 2011 10
Connect. Collaborate. Innovate.
TeamCity

11. © Copyright GlobalLogic 2011 11
Connect. Collaborate. Innovate.
Sample SVG Report

12. © Copyright GlobalLogic 2011 12
Connect. Collaborate. Innovate.
Concurrent performance testing

13. © Copyright GlobalLogic 2011 13
Connect. Collaborate. Innovate.
Conclusion
Input data
• 9 priority-driven configurations
• 2 performance engineers
• Two-week sprints
Completed activities:
• Performance testing process was automated (automatic application deployment,
tests execution, results analysis)
• Test LAB upgraded to support 4 concurrent performance testing executions against
isolated servers (25 servers in the LAB)
Statistics
• 3 months spent on building Jmeter based automated performance testing
• Automated deployment and configuration process is ~25% faster than manual
• Performance testing cycle takes ~4 days (instead of 1+ man-months estimated)
• 50% of time: results analysis, troubleshooting
• 10% of time: keep automation in a good state
• 40% of time: new tests development, exploratory testing, cluster tests
• 17 performance issues were found
• Performance testing reports are iteratively used by PM’s to assess quality of the
product
• PMs are happy 

14. © Copyright GlobalLogic 2011 14
Connect. Collaborate. Innovate.
Q&A
Thank you
skype: andreymelnyk