1. Fundamentals of Application Performance Testing
Practical Performance Analyst – 28th July 2012
Performance Engineering Life Cycle
What is Proactive Performance Management
Holistic View of Performance
What is Performance Testing
Why is Performance Testing Important
Aspects to consider while Performance Testing
Pitfalls to avoid during Performance Testing
Performance Testing Process
Challenges involved in Performance Testing
Deliverables for the Performance Testing process
Resources & tools for Performance Testing
3. Performance Engineering Life Cycle
Software Development Life Cycle
Functional Requirements Gathering
Architecture & Design
System Integrated Test & UAT
Deploy Into Production
Performance Engineering Life Cycle
Non Functional Requirements Gathering
Design for Performance &
Unit Performance Test & Code Optimization
Monitoring & Capacity Management
4. Proactive Performance Management
Performance Requirements Analysis
Performance Modelling & Capacity Planning
Build & Optimization
Proactive Performance Management helps by –
Identifying performance issues early in the design
Validate suitability of your application architecture for your documented NFR’s
Validate your application capacity requirements for the documented NFR’s
Advocates early SVT to identify potential bottlenecks early in the build and development cycle
Reduces potential for major rework later in the program
Advocates using modelling techniques to predict performance impact for growing workload
5. Txn Performance
- Response Times, etc.
Application Performance – Operations/Sec, Messages/Sec, Transactions/Sec, etc.
Infrastructure Performance – CPU Utilization, Memory Utilization, Disk IOPS, etc.
Network Performance – Packet Loss, Jitter, Packet Re- ordering, Delay, etc.
Holistic View of Performance
6. What Is Performance Testing
Performance Testing is quite often confused with Performance Engineering. Although Performance Testing is an integral and important part of Performance Engineering its usefulness is confined to validating Application Performance later in the SDLC (Software Development Life Cycle).
Performance Engineering as we’ve defined it in the past consists of a set of roles, responsibilities, processes and deliverables across the SDLC whose objective is to build applications that meet their NFR’s.
Performance Testing can be defined as the process of validating your application performance against the set of defined Non Functional Requirements.
The objectives of Performance Testing are generally a combination of the following:
Validate Application Scalability – Performance Testing
Validate Application Performance at Peak Workload – Stress Testing
Validate Application Performance over a extended duration of time – Soak Testing
Establish Application Breaking Point – Break Point Testing
Performance Testing has traditionally been and continues to be a proactive process. Programs and project generally lack both the capability and willingness to consider Performance early in the SDLC.
While Performance Testing allows for validation of Application Performance later in the SDLC, a Performance Engineer can use a combination of different modelling techniques to validate application designs, infrastructure capacity and application performance early in the SDLC and mitigate risk accordingly.
7. Why Is Performance Testing Important
Performance Testing is important for the following reasons –
Identify bottlenecks across the application and highlight opportunities for tuning
Validate your design & infrastructure capacity assumptions made early at the design stage
To validate if your applications meet the program Non Functional Requirements
Validate that the application is able to deliver the expected End User Experience while consuming resources optimally across the infrastructure and network tier
To determine the breaking point of your applications. Knowing the breaking point of your applications helps you understand the maximum Business Workload the application can process before performance starts to degrade.
8. Aspects to consider while Performance Testing
Following are the aspect you should keep in mind when planning Performance Testing –
Performance Testing should ideally start through setting up of the program Performance Engineering team at the Requirements Gathering & Design Stage. At this stage you should be working on closing of the following documents:
Program Performance Engineering Strategy
Performance Testing Workload
Non Functional Requirements
Performance Testing Strategy
Start early and don’t wait until the code base completely stabilizes. I encourage you to start Performance Testing early, identify opportunities for tuning and go back with feedback to the development teams.
Work closely with the development teams to meet your documented NFR’s. Make sure you have the buy in of your program leads, environment leads and development team leads. With the appropriate buy in issues get addressed a lot faster.
Discuss and document your environment requirements. Anything lesser than 100% capacity of production will require you to use some black magic (aka. Performance Modelling Techniques) to extrapolate performance characteristics
Discuss and document your data requirements (Script data requirements, application data requirements, etc.). Data preparation can be one of the most challenging tasks on a program.
Discuss & document your Performance Testing and Performance Monitoring (Application + Infrastructure) tool requirements. You will also need to discuss and work out the cost implications due to Virtual Users (VU) licensing.
Make sure everyone the program understands the NFR’s and overall SVT Business Workload very clearly. Ambiguity in any of these areas will land you in serious trouble later in the program.
Track performance issues closely. Use your program risks & issues register to track performance bottlenecks that are preventing you from progressing with SVT.
9. Aspects to consider while Performance Testing
Consider having a good balance of Performance Engineers, Performance Testers, Diagnostics resources & Application Experts. To be successful you should plan to build a Performance Testing team that doesn’t just execute but also knows how to identify relevant defects, tune the application/database and work with the development teams in coming up with fixes to relevant Performance issues.
Plan your resourcing model early on. To obtain a good balance of Performance Engineers & Performance Testers you will need a good head start. Obtaining Performance Engineering skills is generally tough and you might want to look at sourcing them from a niche consulting outfit.
Don’t wait till for SIT & UAT to complete for you to start planning for Performance Test. Performance Testing just before go live is the best way to ensure that your application fails to meet the documented NFR’s (if they even exist i.e.).
10. Pitfalls to avoid during Performance Testing
Ensure that you’ve got well defined and clearly articulated Non Functional Requirements. Your Non Functional Requirements should be signed of by business and program management.
Ensure that you’ve got well defined and clearly articulated Performance Testing Workload & Performance Testing Strategy. Your Workload & Strategy should be signed of by business and program management.
Ensure that you’ve got the appropriate tools for Performance Testing, Diagnostics & Performance Monitoring. Avoid recommending or using Open Source tool as far as Performance Testing is concerned. There are quite a few affordable tools (including cloud based solutions) for Performance Testing.
Push for an Performance Testing environment that’s equivalent to production. This will ensure that you don’t have to extrapolate performance of the application from your testing environment into the production environment.
Model application performance early on in the SDLC using modelling techniques described earlier. You can use a combination of analytical and simulation modelling techniques to validate application performance for the given combination of application architecture and infrastructure capacity.
Update your Business Workload for Performance Testing as your business grows and the environment business works in changes. Change in Business Workload would require updating of your NFR’s and rebase lining of your Workload for Performance Test. It changes your requirements and can have considerable impact on tool requirements, infrastructure requirements, resource requirements, etc. but is an important that that should be tracked through the life of the program.
11. Performance Testing Process
Document Non Functional Requirements
Document Performance Testing Workload
Document Performance Testing Strategy
Document tooling, licensing, data, environment requirements
Document resource requirements
Setup Performance Testing Team
Setup Environment for Performance Testing
Create Performance Testing Scripts
Create Data for Performance Testing
Setup Performance Testing Workload Scenarios within Tesing Tool
Setup Monitoring & Diagnostics tools
Execute Performance Tests
( A )
Collect test data from testing, monitoring & diagnostics tool
Identify performance bottlenecks & tune the application
Exit if we’ve met our NFRs else loop back to Step ( A )
12. Challenges involved in Performance Testing
Challenges convincing the delivery teams to focus on Performance Engineering & Performance Testing activities early in the SDLC
Challenges obtaining funding for the appropriate performance testing, monitoring & diagnostics tools
Challenges obtaining the right combination of resources to address your Performance Testing needs
Challenges obtaining clean, transformed and scaled data for your Performance Testing needs
Challenges getting access to Business SME for duration of Performance Test
Challenges getting access to the application SME’s during the course of the program
Challenges getting buy in from all the development and support teams to support Performance Testing activities
Challenges convincing the client to invest in Performance Testing infrastructure equivalent to production
Challenges due to lack of Non Functional Requirements
Challenges due to lack of understanding of Business workload
13. Deliverables – Performance Testing
Following would be the deliverables for Performance Testing -
Non Functional Requirements document (If not created during the initial PE phase)
Overall Business Workload to be processed by the application (If not created during the initial PE phase)
Overall Performance Testing approach