For Impetus’ White Papers archive, visit- http://www.impetus.com/whitepaper This paper focuses on software performance diagnostic tools and how they enable rapid bottleneck identification.

1. Identifying Software Performance
Bottlenecks using Diagnostic tools
W H I T E P A P E R
Abstract
In this white paper, Impetus Technologies focuses on software
performance diagnostic tools and how they enable rapid
bottleneck identification. It discusses server-side and client-side
diagnostics, including pluggable diagnostic tools and custom
solutions across various scenarios.
The white paper also touches on the benefits of adopting
diagnostics, such as enhanced collaboration and application
visibility.
Impetus Technologies, Inc.
www.impetus.com

2. Identifying Software Performance Bottlenecks using Diagnostic tools
2
Table of Contents
Introduction...........................................................................................................3
The importance of Diagnostic tools.......................................................................3
How diagnostic tools work?...................................................................................5
Impetus’ approach to server-side diagnostics.......................................................7
Impetus’ solution for client side diagnostics .........................................................9
Benefits offered by Diagnostics tools ..................................................................10
Summary..............................................................................................................11

3. Identifying Software Performance Bottlenecks using Diagnostic tools
3
Introduction
Identifying bottlenecks and performance snags in applications is becoming
critical for organizations today. Typically, this is undertaken using load test
results. The performance engineering team detects problem areas by executing
performance tests, and the test results, including problem areas, transaction
response times, and system monitoring data are then shared with the
development team. The development team identifies system bottlenecks using
resource monitoring data and helps in the tuning of system components. These
recommendations are then shared with the performance engineering team,
that validates the same with performance tests.
However, identifying code level bottlenecks is a challenging task for the
development team. It may find it difficult to analyze and comprehend the test
results, as they are based on user transactions. Also, development
environments are vastly different from test environments in terms of memory
and processing power.
Organizations are therefore on the lookout for solutions that can help them
reduce the iterations between the performance engineers and the development
teams and speed up the process of identifying and isolating code bottlenecks.
They are keen to find ways by which they can map user transaction results to
the method and query level, so that the development team can be helped in its
analysis. Diagnostics tools can help companies address these issues.
The Importance of Diagnostic Tools
Diagnostic tools map the user-centric business transactions to method calls,
thereby providing a developer-centric view of the test results. These tools are
beneficial for the following reasons:
• They can drill down the transaction response time into underlying
methods, queries and service calls to components; and provide
response time break-down across different sub-components.
• They follow the path of the user transaction and gather performance
data throughout its lifetime.
• They enable the development team to analyze these statistics along
with the method trace, and isolate the bottlenecks. This provides an
intuitive, easy-to-use view, of how individual tiers, components,
memory, and SQL statements impact the overall performance of a
business process under load conditions; and facilitates deeper analysis.

4. Identifying Software Performance Bottlenecks using Diagnostic tools
4
• They can be integrated with the available load testing solutions and
effectively used in test environments.
• They can be used in production to set up a continuous performance
monitoring mechanism, due to their low overheads.
However it is important to differentiate between the terms ‘diagnostic tools,’
and ‘profilers,’ which are often used by teams interchangeably. The fact is that
these are two different sets of tools with different purposes.
Profilers are more suited for fine-grained analysis and usually have more
overheads than diagnostic tools. This makes them inappropriate for multi-user
tests. Diagnostic tools on the other hand, can be used for multi-user load tests,
as well as, production monitoring. Their main purpose is to isolate the code
bottlenecks and provide relevant information. The development team can use
Profilers to explore these isolated problem areas and figure out their root cause.
Essentially, Profilers and Diagnostic tools complement each other.

5. Identifying Software Performance Bottlenecks using Diagnostic tools
5
How Diagnostic Tools Work?
Diagnostic tools can be used in multiple environments including development,
test, staging and production. However, they are most widely used in a load-
testing environment.
Figure 1 How Diagnostic Tools Work
Diagnostics can be applied to the app server, as well as the client-side, including
the browser and mobile device. Diagnostic tools have two major components—
the Diagnostic agent and Diagnostic server.
The Diagnostic agent runs on the target component and hooks up with it, using
byte code instrumentation. It gathers performance statistics and starts
transmitting the data to the server. The agent runs in the same runtime as that
of the target component.

6. Identifying Software Performance Bottlenecks using Diagnostic tools
6
The Diagnostic server is responsible for collecting the raw data, aggregating it
and storing it in the result repository. The user can then generate multiple
graphs and reports; and monitor the application performance in real time as
well as perform an offline analysis.
Server side diagnostics
There are different scenarios where diagnostic tools can be used. Key among
them is server side diagnostics. In this particular scenario, Diagnostic tools are
used to identify critical issues in the server components. These may include the
application server, database server, Web server, messaging server, among
others. The performance engineering team can choose between plug-in tools or
custom tools.
Plug-in Diagnostic tools
Plug-in Diagnostic tools are integrated with the load testing tool. There are also
situations where companies have to create a custom diagnostic harness to
support some specific technology or requirement. These tools do not integrate
with any load testing tool and work in isolation.
Plug-in tools are used as an extension to existing load testing tools. Their major
advantage is that they can integrate diagnostic data with test results. The
business processes, in this case, can be mapped to method and query calls.
Depending on the path of the user transaction in the application, the
distribution of response time is provided.
This makes analysis easier for the development team. The team can not only
identify the method execution times but also verify any redundant method calls.
The diagnostics configuration is handled by the load testing tool. The methods
that need to be monitored are included as a part of the test scripts that are
executed.
Custom Diagnostic tools
Standalone Diagnostic tools work independently, monitoring the performance
statistics of the target application, as well as mapping the method and query
details to the request URLs as received by the server component.
These tools are installed and configured separately in the target environment.
They are well suited to work with existing load testing tools and test scripts.
Depending on the technology requirements, a custom tool might be created
that can work during the load tests and provide performance monitoring.
There are a few Open Source utilities that are also available for the same.
Glassbox, and Infrared are some of popular tools in this category.

7. Identifying Software Performance Bottlenecks using Diagnostic tools
7
Impetus’ Approach to Server-Side Diagnostics
Impetus Technologies has evolved an approach to server-side diagnostics that it
has used effectively in multiple customer engagements.
The first and the most important step are selecting the appropriate Diagnostic
tool. The choice is of course between a plug-in tool or creating a custom
solution, which in turn depends on several factors, including the following:
• The technology being used. If no existing tool is available for the
technology stack being used in the application, it becomes imperative to
create a custom solution using available Aspect oriented frameworks.
• The objective of using Diagnostic tools. If the objective is to use these
during load testing activities, organizations can choose either plug-in or
custom tools. However, if the objective is to continuously monitor
performance, there is need for standalone tools.
• The budget. If the budget is a constraint, organizations can opt for
Open Source or freely available tools. Else, they can go for specialized
tools that provide rich analysis capabilities and are more user-friendly.
After selecting the tool and configuring it in the target environment, Impetus
recommends that performance teams measure the overhead in terms of CPU
utilization and impact on response times. It is possible to control this overhead
by setting up the appropriate filters so that only the specific methods can be
monitored. This is very important in case of the production monitoring
mechanism because companies may not want to have a high impact of the
Diagnostic tool on the environment.
After the tool is properly configured and calibrated, it is possible to run load
tests with the diagnostics on. The test results along with the diagnostic results
can then be analyzed to get insights on the performance of the complete
application.
This includes information such as top 10 methods by execution time, method
hot spots, costliest queries, etc.
Client side diagnostics
Diagnostic tools can also be used to identify client side performance snags,
covering desktop clients, such as browsers; and the apps running on mobile
devices. This is extremely important as client side performance greatly impacts
the end user experience.

8. Identifying Software Performance Bottlenecks using Diagnostic tools
8
Mobile Diagnostics
Mobile diagnostics refers to the measurement of the performance of a mobile
application on a device. This is a challenging task for performance engineers.
Due to the lack of resources on the device, the performance of an application
can degrade drastically. Also, the application may behave differently on
different devices, as well as technologies.
By using diagnostic tools, performance engineering teams can arrive at method
execution times on the mobile device and log the same. This may help in
analyzing the expensive methods.
There aren’t many diagnostic tools available in the mobile space. Profilers,
including J2ME profilers, Android Trace, etc. have to be used, which are specific
to different technologies and are mostly deployed by the development teams.
However, in order to address specific requirements, it is preferable that teams
create custom diagnostic solutions.
Browser Diagnostics
Internet technologies running applications also face issues. Today, rich Internet
applications are available that have been built using AJAX, HTML5, FLEX,
SilverLight, jQuery and similar technologies. These also present performance
challenges to Web developers. It is really difficult to identify bottlenecks on the
browser that can impact the end-user experience. Diagnostic tools can help in
this process.
Most of the client-side diagnostic tools are plug-ins to popular browsers. These
can be integrated with the browser and provide all the statistics related to page
rendering, Java script executions, memory, CPU utilization, etc. Some of the
popular diagnostic plug-ins includes Firebug for Mozilla, Speedtrace for Chrome,
HttpWatch for IE.
There are other set of tools, including Dynatrace Ajax Edition, that work
independently and extract the above information from the browser. These have
more features and provide dashboards and fine grained analysis for identifying
any existing issues.

9. Identifying Software Performance Bottlenecks using Diagnostic tools
9
Impetus’ Solution for Client Side Diagnostics
Not too many tools are available for mobile diagnostics, as multiple
technologies and platforms exist, and no single tool can serve the purpose. It is
better to create custom diagnostic solutions that address specific requirements
of mobile applications.
In few engagements, Impetus has used Aspect-oriented frameworks to build a
diagnostic utility for mobile applications on the devices. The AOP frameworks
like AspectMe for J2ME and AspectLUA for LUA can be used for this purpose.
These frameworks use byte code instrumentation to inject hooks into the target
application and measure runtime statistics, such as method timing and memory
utilization. A set of re-usable aspects can be created that can be hooked into the
important method calls or UI rendering calls to track their performance
statistics.
The aspects can be inserted using application binaries and hence there is no
need to undertake any code changes. The methods to be monitored are
specified at the time of instrumentation.
In order to perform diagnostics for browsers to identify performance
bottlenecks, we recommend installing proper plug-ins and configuring these to
collect performance statistics during browsing session.
The plug-in gathers all the relevant details and makes them available to the
performance testing team. The team can then analyze the results to identify
bottlenecks related to page loading, Java script execution, memory leaks, CPU
utilization, etc.
In one of our engagement by just tuning client side components, application
performance, especially the end user experience, was improved by almost 40%.

10. Identifying Software Performance Bottlenecks using Diagnostic tools
10
Benefits Offered by Diagnostics Tools
Diagnostic tools offer a range of benefits to performance testing and
development teams within organizations, including the following:
• Identification and isolation of performance bottlenecks, which may be
components, methods, or SQL statements. This in turn, assists in
resolving memory, exception, or other problems swiftly, thereby,
reducing the efforts of the development team.
• Complete application visibility across the lifecycle, ensuring higher
quality when applications go live. They also help reduce the Mean Time
to Resolution in the application deployment environment.
• Documentation of the root cause of any performance or stability issues,
including the offending line of code, even in complex, highly distributed
applications. The performance testers do not need any programming
knowledge or access to source code to deliver an offline diagnosis
report that allows development to efficiently resolve issues.
• Effective collaboration between performance engineers and the
development team, due to the availability of detailed test results. This
reduces the multiple iterations between both the teams and speeds up
the problem resolution time.
• Improvement in the productivity of the Development team, including
developers, architects and database engineers. The time-intensive,
tedious, error prone and often impossible process of manually
correlating log files, from different back-end systems with the client-
side load testing results, becomes obsolete. This significantly
accelerates the problem resolution time.
• Inbuilt intelligence and suggestions for teams based on best practices.
Also, some of the tools can be directly integrated with the developer’s
environment and pinpoint the bottleneck at the code level itself.

11. Identifying Software Performance Bottlenecks using Diagnostic tools
11
Summary
In summary, it can be said that Diagnostic tools can do a lot to improve
application performance, by identifying the bottlenecks. Also, companies must
learn to differentiate between Diagnostic tools and Profilers that are not
interchangeable, but rather complementary to each other. Profilers have high
overheads and are better suited for fine-grained analysis.
Organizations can use Diagnostic tools in multiple environments including
development, test, staging and production. It is very essential for companies to
embrace Diagnostics in their performance engineering strategies to enhance
visibility across applications and improve collaboration among various
stakeholders.
It is possible to build custom diagnostics solutions to address specific
requirements. These can be installed and configured separately in the target
environment and are well suited to work with existing load testing tools and test
scripts.
It is also advisable to minimize the overheads by applying filters on the
methods, or URLs that have to be monitored. It is a good practice to calibrate
the overheads of the Diagnostic tools in the test environment by running a few
sanity tests
About Impetus
Impetus is a Software Solutions and Services Company with deep
technical maturity that brings you thought leadership, proactive
innovation, and a track record of success. Our Services and Solutions
portfolio includes Carrier grade large systems, Big Data, Cloud,
Enterprise Mobility, and Test and Performance Engineering.
Website: www.impetus.com | Email: inquiry@impetus.com
© 2013 Impetus Technologies, Inc.
All rights reserved. Product and
company names mentioned herein
may be trademarks of their
respective companies. May 2013