Suddenly, application performance is important to your business, and you have been given the budget to improve it. You’re in a hurry because customers are complaining or because you expect jumps in transaction volume and your application needs to scale quickly. Do you know where to start? Join Ryan Riehle as he shares his experiences developing enterprise performance testing programs. Ryan covers the key techniques and heuristics that lead to an effective performance improvement effort. He discusses patterns teams use to effectively collaborate to achieve performance requirements, how to configure and organize test environments, considerations for application deployment and release cycles, appropriate metrics to use and how to report them, and strategies and techniques for data movement that support reproducible test results. But measuring alone does not solve the performance problem. So Ryan discusses how teams can act on testing results to improve and verify the impact of application and infrastructure changes.

1. 3/18/2015
Copyright 2015 - InCycle Software 1
Implement an Enterprise
Performance Test Process
Ryan Riehle
Senior Consultant, ALM Practice
InCycle Software
© InCycle Software - 2015
Lessons learned along the way
• Improve application performance incrementally while
• Delivering new valuable features / enhancements
• Finding (new) issues close to when they are created
• Focus on client server & web applications – mostly
• Business process and technology practices
© InCycle Software - 2015

2. 3/18/2015
Copyright 2015 - InCycle Software 2
Making a business case
Quantify the value of performance improvements to momentum and lasting
success.
© InCycle Software - 2015
Common Scenarios
• Strategic / Competitive
• Risk Reduction
• Operational
• Morale
© InCycle Software - 2015

3. 3/18/2015
Copyright 2015 - InCycle Software 3
Key roles in the process
• Business (Product Owner / BA)
• Project Manager
• Dedicated Performance Testers?
• Developers
• QA Manager
• User Experience (UX) Designers
• IT Infrastructure Engineers
• Database Administrators
• Vendors
• End Users
© InCycle Software - 2015
Skills of the Performance Tester role
• Platform Configuration and Deployment
• Scripting / Programming
• Tooling
• OS
• Hardware knowledge
• Networking
• Understanding of statistics and variability
• Databases / SQL
© InCycle Software - 2015

4. 3/18/2015
Copyright 2015 - InCycle Software 4
What are we looking for?
Teams must be able to clearly identify and communicate performance goals and
requirements and be consistent about how performance fits in.
© InCycle Software - 2015
Example
© InCycle Software - 2015

5. 3/18/2015
Copyright 2015 - InCycle Software 5
Prioritizing Performance
• Focus on immediate value of improvement
• Value to the business and stakeholders (cost / efficiency / ROI)
• Value to the end-users
• Look for low hanging fruit
• Bottlenecks are opportunities
• Technical debt
© InCycle Software - 2015
Perception of Performance
• Commonly accepted guidelines for perceived performance
• 100ms – immediate
• 24-60 Frames per second FPS ~20ms (for smooth video)
• 100ms to 1000ms – noticeable delay
• >1s – mental context switching
• >10 seconds – abandonment
• Just Noticeable Difference
• For application performance this is generally accepted at about 20%
© InCycle Software - 2015

6. 3/18/2015
Copyright 2015 - InCycle Software 6
Performance standards become
the rule
Exceptions can be noted in the requirements
© InCycle Software - 2015
Example
© InCycle Software - 2015

7. 3/18/2015
Copyright 2015 - InCycle Software 7
Document your performance test
process
© InCycle Software - 2015
Tools and Technology Considerations
• Types of applications and components
• Web, Web service, Cloud, Mobile, etc…
• Programming languages
• Technology culture and conventions
• Operating systems
• Standardization
• Internal vs Public facing
• Database Technologies
• Build and deployment practices and supporting technologies
• Available skills and resources
• Audience of reports
• Audit Requirements
• Network and security
© InCycle Software - 2015

8. 3/18/2015
Copyright 2015 - InCycle Software 8
Provision a performance testing environment
• Ideally a copy of production environments but frequently not possible
• Match As closely as possible, then correlate performance
• Don’t virtualize non-virtualized servers for performance testing
• Security – testers and developers will need access to monitor
performance metrics.
• Tooling may also require service accounts and firewall changes
• Anti-virus and system management software can have a significant
impact.
© InCycle Software - 2015
Start automating the deployment
Production and test environments’ configuration settings need to match – a single
wrong setting could mean meaningless performance test results.
© InCycle Software - 2015

9. 3/18/2015
Copyright 2015 - InCycle Software 9
Plan to generate load reliably
Dedicate machines for load generation, don’t do it from Joe’s laptop.
© InCycle Software - 2015
Minimize variables impacting test
results like network bandwidth
and geography
Isolate the primary application and introduce new variables iteratively if
appropriate.
© InCycle Software - 2015

10. 3/18/2015
Copyright 2015 - InCycle Software 10
Fake it until you make it
You don’t need to be limited by dependencies and external systems
© InCycle Software - 2015
Create Baseline Tests
Performance tests need to be repeatable so results can be compared.
© InCycle Software - 2015

11. 3/18/2015
Copyright 2015 - InCycle Software 11
Test runs can take time to get
right
Find a balance between how quickly load is applied and the length of the test.
© InCycle Software - 2015
Automate performance testing
Aim to automate from the beginning so that every time you build and deploy you
have performance test results.
© InCycle Software - 2015

12. 3/18/2015
Copyright 2015 - InCycle Software 12
Desktop applications can be
tested by automating the UI
Generate load on the server using calls the client application would use or else it
will be expensive to automate the use of most desktop apps on multiple computers
simultaneously.
© InCycle Software - 2015
Plan Data Movement for
Reproducible Results
© InCycle Software - 2015

13. 3/18/2015
Copyright 2015 - InCycle Software 13
Role of the developer
• Refactoring
• Concurrency testing
• Knowledge
• Small loads
• Automation
• Build
• Data
• Architecture
© InCycle Software - 2015
Collaboration Patterns for Dev Teams
• Source control practices
• Keep requirements and tests easy to understand
• Automation as documentation
• Ideally, performance testers are part of the development team
© InCycle Software - 2015

14. 3/18/2015
Copyright 2015 - InCycle Software 14
A Focus on Core Metrics
• Metrics
• CPU time
• I/O
• Memory usage
• NIC / Network performance
© InCycle Software - 2015
Reporting Practices
• Include context
• Test environment configuration
• Test scenario – what was performed during the test
• Report visually with graphs and annotations
• Consider separate reports for business and technology
• Create actionable bugs or tasks in your tracking system to represent
problems identified and track them through to completion.
• Record video if possible to show business and developer
© InCycle Software - 2015