Automated
performance testing
using Maven & JMeter
George Barnett, Atlassian Software Systems
@georgebarnett

• Create controllable JMeter tests
• Configure Maven to create a repeatable cycle
• Run this “build” in your CI server every 8 hours
• Find performance regressions faster

WHY?
• Catch regressions quickly
• Stop hunting for bugs
• Code with confidence
• Know the performance cost of features
• Fearless external dependency upgrades
• Go home on time

ATLASSIAN
• 15,000+ customers
• 100+ developers
• 8 software streams
• 1 performance engineer

ATLASSIAN
• JIRA - Issue management
• Confluence - Enterprise Wiki
• Bamboo - Continuous Integration
• FishEye, Crucible & Clover - Dev Tools
• Crowd - Single Sign-On.

BEFORE WE BEGIN
• Continuous Integration is critical
• Keep code working with Unit and Functional
tests
• Broken or irregular builds means no code to
performance test
• Performance testing is NOT a replacement
for other testing (but it helps)

PROCESS
• Write code (1-6 months)
• Compile code
• Send it to QA (or Production?!)
• Performance Regression
• PANIC!
No performance testing!

A BETTER WAY
• Write code
• CI compiles code and runs tests
• Send it to QA (or Production?!)
• Performance Regression
• PANIC!
No performance testing!

A BETTER WAY 2.0
• Performance test artifacts as soon as they
pass Unit and Functional testing
• Fast feedback for developers
• No Surprises

TOOLS
• JMeter
• Maven & plugins
• Bamboo & plugins
• A Profiler

JMETER
• Apache Project
• Java
• Supports HTTP, JDBC, SOAP, XML-RPC,
FTP, SMTP, LDAP, JUnit & more.

MAVEN
• Apache Project
• Java
• Build Tool
• Dependency management
• Documentation

MAVEN CONCEPTS
• Project Object Model (POM)
• Standardised directory structure
• Build Lifecycle
• Reporting Lifecycle
• Plugins

MAVEN POM
<project>
<groupid>com.atlassian.com.performance.jira</groupid>
<artifactid>automated-performance</artifactid>
<properties>
<test.dataset>/opt/dataset/dataset1.zip</test.dataset>
</properties>
<build>
<!--Main Lifecycle plugins configured here-->
</build>
<reporting>
<!--Reporting Lifecycle here-->
</reporting>
</project>

MAVEN PLUGINS
• Chronos
• Ant-run
• Cargo
• Maven Assembly plugin
• Maven Dependency plugin

BAMBOO
• Continuous Integration
• Can run builds on Amazon’s EC2
• JMeter Aggregator Plugin

A PROFILER
• Java Profiler
• CPU Snapshots
• GC & Memory information
• Thread usage and contention
• Triggers & offline sampling

STEP 1
Package the JMeter test
• Maven Assembly plugin
• Create a .zip from project directories
• `mvn deploy` to repository as a SNAPSHOT

STEP 2
Configure the Maven automated build
• Package application state into zip files
• Ant-run - application pre-setup
• Cargo - deploy container
• Chronos - configure the JMeter tests

MAVEN PROFILES
<project>
..
<properties>
<test.dataset>/opt/dataset/dataset1.zip</test.dataset>
</properties>
<profile>
<id>dataset1</id>
<properties>
<test.dataset>/opt/dataset/dataset1.zip</test.dataset>
..
<profile>
<id>dataset2</id>
<properties>
<test.dataset>/opt/dataset/dataset2.zip</test.dataset>
..

MAVEN PROFILES
• Invoked on command line
• eg: mvn verify -Pmysql, tomcat5, dataset1
• Configuration in profile is added to build
• Use for a wider variation in test
environments
• Use profiles for individual “work units”, eg
JMeter

ADDING A PROFILER
$ export LD_LIBRARY_PATH=”$JPHOME/bin/linux-x64/
<global.jvm.args>-Xms2000m -Xloggc:${project.build.directory}/gc.log</global.jvm.args>
<jvm.args>${global.jvm.args}</jvm.args>
<profile>
<id>jprofiler</id>
<properties>
<jvm.args>-agentlib:jprofilerti=offline,id=${jprofiler.profileid},config=${jprofiler.configfile} -
Xbootclasspath/a:${jprofiler.home}/bin/agent.jar ${global.jvm.args}</jvm.args>
</properties>
</profile>

REPORTING
CHRONOS
• Available in maven
• eg: mvn site / mvn chronos:report
• Generates graphs and a numerical report
• Can compare to historical data
• Static :(

REPORTING
CHRONOS

REPORTING
CHRONOS

REPORTING
CHRONOS

REPORTING
CHRONOS

REPORTING
CHRONOS - HISTORY

REPORTING
CHRONOS - HISTORY

REPORTING
BAMBOO JMETER AGGREGATOR
• Open source Bamboo plugin (supported!)
• James Roper (Atlassian) 20% time project
• Can graph JMeter .JTL and CSV files
• Dynamic

REPORTING
BAMBOO JMETER AGGREGATOR

TIPS
• Allocate time for fixing tests
• Your tests WILL break
• Work out a process for fixing bugs
• Gather as many artifacts from each test run
as possible
• React sooner rather than later

Q&A
gbarnett @ atlassian . com
@georgebarnett
Code samples online soon!
blogs.atlassian.com/developer
Meet Atlassian @ The Summit:
May 31 - June 2 in SF
www.atlassian.com/summit