Traditional large scale end-of-cycle performance tests served enterprises well in the waterfall era. However, as organizations transition to agile development models, many find their tried and true approach to performance testing—and their performance testing resources—becoming somewhat irrelevant. The strict requirements and lengthy durations just don’t fit in the context of an agile cycle. Additionally, investigating system performance at the end of the development effort misses out on the early stage feedback offered by an agile approach. And it’s more important than ever that today’s agile-built systems perform. So how can agile organizations ensure optimum performance of their business critical systems? Lee Barnes discusses why agile teams need to change their thinking about performance from a narrow focus on testing to a broader focus on analysis—from a people, process and technology perspective. Take back techniques for shifting your performance testing/analysis earlier in the development cycle and extracting performance data that is immediately actionable.

1. © Utopia Solutions
Performance Testing in the Agile Lifecycle
Lee Barnes, CTO
Utopia Solutions
@USI_LeeBarnes

2. © Utopia Solutions2
Key Questions
What needs to change?
How do we make those changes?
Why don’t traditional performance
testing models fit nicely with agile?

3. © Utopia Solutions3
Case Study
Back story…
• Large global manufacturer
• Performance Testing CoE in operation for ~10 yrs
• Seen as a huge value to the enterprise
• Began moving toward agile a few years ago…
and the luster of the CoE began to dull…

4. © Utopia Solutions4
Traditional Performance Testing Model
Center of
Excellence
Project 1
Project 2 Project
N
Provides Consistency in…
• Resources / Skills
• Processes
• Deliverable Quality
• Technology
Request Test
Project
Team
Center of
Excellence
Perform Discovery
Develop & Execute
Deliver ResultsEvaluate Results
Info / Environment

5. © Utopia Solutions5
CoE in the Waterfall Years
Plan Design Build Test Deploy
• One size fits all
• Strict entry criteria
• Performed late in SDLC
• One-way communication of
results
Service Characteristics
Performance Testing
Too late!

6. © Utopia Solutions6
Agile Development
Analyze
Develop
Integrate
Acceptance
Test
Deploy
Where does performance
testing fit?

7. © Utopia Solutions7
Early Attempts on Agile Projects
Sprint 1 Sprint 2 Sprint 3 … Deploy
Agile
Performance Testing
Same Process…
Same Risk!

8. © Utopia Solutions8
Early Attempts on Agile Projects
Sprint 1 Sprint 2 Sprint 3 … Deploy
Agile
Performance
Testing
Performance
Testing
Performance
Testing
Performance
Testing
CoE Approach
• Rigid
• 4 week test cycle
Agile Approach
• Flexible
• 2 week sprints
+ =

9. © Utopia Solutions
Back to the drawing
board…

10. © Utopia Solutions10
Early Cycle /
Ongoing
Performance
Analysis
Establish Objectives
Ultimate Goal…
Late Cycle
Performance
Testing
Shift Left

11. © Utopia Solutions11
Identify Gaps
What’s stopping us from analyzing performance
continuously?
• Performance objectives weren’t part of
user stories
• Agile teams lacked performance
engineering expertise
• APM tools / skills were non-existent

12. © Utopia Solutions12
Breakdown the Barrier Between CoE and Agile Teams

13. © Utopia Solutions13
Disclaimers
It was not easy.
The changes discussed on the next few
slides were implemented over a two
year duration.
It was not painless.

14. © Utopia Solutions14
Performance Testing in an Agile Environment
Capture non-functional requirements
Make performance everyone’s job
Create a role to coordinate
performance activities
CoE performance test architects
become mentors
Begin developing a
performance focused
culture

15. © Utopia Solutions15
Incorporating Performance Requirements…
… as acceptance criteria for existing user stories
As a dealer I want to create a
product configuration for an order so
that my customer can purchase the
equipment they need
• All steps in the product config.
process will have a response time
of less than 2 seconds
• 1,500 dealers are concurrently
using the system to configure
products for customer orders
… as technical stories
As a dealer, I want the response time of any step in the product
configuration process to be less than 2 seconds
… as constraints
The system will respond in less than 2 seconds for all steps in the
product configuration process for a maximum of 1,500 concurrent users
configuring orders

16. © Utopia Solutions16
Performance Testing in an Agile Environment
Unit performance testing
Component / system level performance
testing
Include performance testing in the CI
chain
Incorporate performance
testing / analysis into
each sprint

17. © Utopia Solutions17
Performance Unit Testing
Start with existing unit tests
“Decorate” them with
timers and load
Monitor results across
builds
Build 1 Build 2 Build 3 Build 4

18. © Utopia Solutions18
Beyond Unit Testing
Map user stories onto system
components
Benchmark in development / QA
environment
Look for trends in measurements vs.
absolutes
Use service virtualization for 3rd party
or yet to be developed components
Incorporate as part of the CI chain
Harden scripts against application
changes
Component and
system level testing

19. © Utopia Solutions19
Adding Performance Tests to CI
Lessons Learned
Custom plug-ins exist to run performance
tests as part of the CI chain – very useful
People get mad when you unnecessarily
break the build
They get really mad when you can’t
figure out why you broke the build
Add error handling and logging to
scripts to enable quick failure
investigation
Allow for non-production-like
environment
Set your response time tolerances
and client timeouts accordingly

20. © Utopia Solutions20
Waterfall Agile
Responsibility for
Performance
Performance CoE Everyone
Role of CoE End of cycle PT Provide mentorship to agile team
Pre-deployment system PT
Role of
development
Throw system over wall
Wait for results
Continuous focus on performance
Tools Traditional load test tools APM tools
Service virtualization
Continuous integration
Traditional load test tools
Environment Dedicated / production-like Development / desktop
Deliverable Boiler plate results delivered
just prior to release
Continuous feed of actionable
data
Waterfall Agile
Responsibility for
Performance
Performance CoE
Role of CoE End of cycle PT
Role of
development
Throw system over wall
Wait for results
Tools Traditional load test tools
Environment Dedicated / production-like
Deliverable Boiler plate results delivered
just prior to release
Performance Testing in an Agile Environment
Transformation Summary

21. © Utopia Solutions21
Summary
Key takeaways…
• Agile performance testing is not more
/ quicker performance tests it’s a
continuous focus on performance
• Performance is part of everyone’s job
• Success requires a shift in people,
process and technology

22. © Utopia Solutions22
Questions…
…. and Answers!
Direct future questions to:
Lee Barnes
Founder and CTO
Utopia Solutions, Inc.
Email: lee.barnes@utopiasolutions.com
Twitter: twitter.com/USI_LeeBarnes
LinkedIn: linkedin.com/in/leebarnes
Blog: www.RaiseYourMQ.com
?

23. © Utopia Solutions
Links to Unit Performance Testing Frameworks
• JUnitPerf – http://www.clarkware.com/software/JUnitPerf.html
• P-unit – http://p-unit.sourceforge.net/