Identifying Software Performance Bottlenecks Using Diagnostic Tools- Impetus White Paper
Identifying Software PerformanceBottlenecks using Diagnostic toolsW H I T E P A P E RAbstractIn this white paper, Impetus Technologies focuses on softwareperformance diagnostic tools and how they enable rapidbottleneck identification. It discusses server-side and client-sidediagnostics, including pluggable diagnostic tools and customsolutions across various scenarios.The white paper also touches on the benefits of adoptingdiagnostics, such as enhanced collaboration and applicationvisibility.Impetus Technologies, Inc.www.impetus.com
Identifying Software Performance Bottlenecks using Diagnostic tools2Table of ContentsIntroduction...........................................................................................................3The importance of Diagnostic tools.......................................................................3How diagnostic tools work?...................................................................................5Impetus’ approach to server-side diagnostics.......................................................7Impetus’ solution for client side diagnostics .........................................................9Benefits offered by Diagnostics tools ..................................................................10Summary..............................................................................................................11
Identifying Software Performance Bottlenecks using Diagnostic tools3IntroductionIdentifying bottlenecks and performance snags in applications is becomingcritical for organizations today. Typically, this is undertaken using load testresults. The performance engineering team detects problem areas by executingperformance tests, and the test results, including problem areas, transactionresponse times, and system monitoring data are then shared with thedevelopment team. The development team identifies system bottlenecks usingresource monitoring data and helps in the tuning of system components. Theserecommendations are then shared with the performance engineering team,that validates the same with performance tests.However, identifying code level bottlenecks is a challenging task for thedevelopment team. It may find it difficult to analyze and comprehend the testresults, as they are based on user transactions. Also, developmentenvironments are vastly different from test environments in terms of memoryand processing power.Organizations are therefore on the lookout for solutions that can help themreduce the iterations between the performance engineers and the developmentteams and speed up the process of identifying and isolating code bottlenecks.They are keen to find ways by which they can map user transaction results tothe method and query level, so that the development team can be helped in itsanalysis. Diagnostics tools can help companies address these issues.The Importance of Diagnostic ToolsDiagnostic tools map the user-centric business transactions to method calls,thereby providing a developer-centric view of the test results. These tools arebeneficial for the following reasons:• They can drill down the transaction response time into underlyingmethods, queries and service calls to components; and provideresponse time break-down across different sub-components.• They follow the path of the user transaction and gather performancedata throughout its lifetime.• They enable the development team to analyze these statistics alongwith the method trace, and isolate the bottlenecks. This provides anintuitive, easy-to-use view, of how individual tiers, components,memory, and SQL statements impact the overall performance of abusiness process under load conditions; and facilitates deeper analysis.
Identifying Software Performance Bottlenecks using Diagnostic tools4• They can be integrated with the available load testing solutions andeffectively used in test environments.• They can be used in production to set up a continuous performancemonitoring mechanism, due to their low overheads.However it is important to differentiate between the terms ‘diagnostic tools,’and ‘profilers,’ which are often used by teams interchangeably. The fact is thatthese are two different sets of tools with different purposes.Profilers are more suited for fine-grained analysis and usually have moreoverheads than diagnostic tools. This makes them inappropriate for multi-usertests. Diagnostic tools on the other hand, can be used for multi-user load tests,as well as, production monitoring. Their main purpose is to isolate the codebottlenecks and provide relevant information. The development team can useProfilers to explore these isolated problem areas and figure out their root cause.Essentially, Profilers and Diagnostic tools complement each other.
Identifying Software Performance Bottlenecks using Diagnostic tools5How Diagnostic Tools Work?Diagnostic tools can be used in multiple environments including development,test, staging and production. However, they are most widely used in a load-testing environment.Figure 1 How Diagnostic Tools WorkDiagnostics can be applied to the app server, as well as the client-side, includingthe browser and mobile device. Diagnostic tools have two major components—the Diagnostic agent and Diagnostic server.The Diagnostic agent runs on the target component and hooks up with it, usingbyte code instrumentation. It gathers performance statistics and startstransmitting the data to the server. The agent runs in the same runtime as thatof the target component.
Identifying Software Performance Bottlenecks using Diagnostic tools6The Diagnostic server is responsible for collecting the raw data, aggregating itand storing it in the result repository. The user can then generate multiplegraphs and reports; and monitor the application performance in real time aswell as perform an offline analysis.Server side diagnosticsThere are different scenarios where diagnostic tools can be used. Key amongthem is server side diagnostics. In this particular scenario, Diagnostic tools areused to identify critical issues in the server components. These may include theapplication server, database server, Web server, messaging server, amongothers. The performance engineering team can choose between plug-in tools orcustom tools.Plug-in Diagnostic toolsPlug-in Diagnostic tools are integrated with the load testing tool. There are alsosituations where companies have to create a custom diagnostic harness tosupport some specific technology or requirement. These tools do not integratewith any load testing tool and work in isolation.Plug-in tools are used as an extension to existing load testing tools. Their majoradvantage is that they can integrate diagnostic data with test results. Thebusiness processes, in this case, can be mapped to method and query calls.Depending on the path of the user transaction in the application, thedistribution of response time is provided.This makes analysis easier for the development team. The team can not onlyidentify the method execution times but also verify any redundant method calls.The diagnostics configuration is handled by the load testing tool. The methodsthat need to be monitored are included as a part of the test scripts that areexecuted.Custom Diagnostic toolsStandalone Diagnostic tools work independently, monitoring the performancestatistics of the target application, as well as mapping the method and querydetails to the request URLs as received by the server component.These tools are installed and configured separately in the target environment.They are well suited to work with existing load testing tools and test scripts.Depending on the technology requirements, a custom tool might be createdthat can work during the load tests and provide performance monitoring.There are a few Open Source utilities that are also available for the same.Glassbox, and Infrared are some of popular tools in this category.
Identifying Software Performance Bottlenecks using Diagnostic tools7Impetus’ Approach to Server-Side DiagnosticsImpetus Technologies has evolved an approach to server-side diagnostics that ithas used effectively in multiple customer engagements.The first and the most important step are selecting the appropriate Diagnostictool. The choice is of course between a plug-in tool or creating a customsolution, which in turn depends on several factors, including the following:• The technology being used. If no existing tool is available for thetechnology stack being used in the application, it becomes imperative tocreate a custom solution using available Aspect oriented frameworks.• The objective of using Diagnostic tools. If the objective is to use theseduring load testing activities, organizations can choose either plug-in orcustom tools. However, if the objective is to continuously monitorperformance, there is need for standalone tools.• The budget. If the budget is a constraint, organizations can opt forOpen Source or freely available tools. Else, they can go for specializedtools that provide rich analysis capabilities and are more user-friendly.After selecting the tool and configuring it in the target environment, Impetusrecommends that performance teams measure the overhead in terms of CPUutilization and impact on response times. It is possible to control this overheadby setting up the appropriate filters so that only the specific methods can bemonitored. This is very important in case of the production monitoringmechanism because companies may not want to have a high impact of theDiagnostic tool on the environment.After the tool is properly configured and calibrated, it is possible to run loadtests with the diagnostics on. The test results along with the diagnostic resultscan then be analyzed to get insights on the performance of the completeapplication.This includes information such as top 10 methods by execution time, methodhot spots, costliest queries, etc.Client side diagnosticsDiagnostic tools can also be used to identify client side performance snags,covering desktop clients, such as browsers; and the apps running on mobiledevices. This is extremely important as client side performance greatly impactsthe end user experience.
Identifying Software Performance Bottlenecks using Diagnostic tools8Mobile DiagnosticsMobile diagnostics refers to the measurement of the performance of a mobileapplication on a device. This is a challenging task for performance engineers.Due to the lack of resources on the device, the performance of an applicationcan degrade drastically. Also, the application may behave differently ondifferent devices, as well as technologies.By using diagnostic tools, performance engineering teams can arrive at methodexecution times on the mobile device and log the same. This may help inanalyzing the expensive methods.There aren’t many diagnostic tools available in the mobile space. Profilers,including J2ME profilers, Android Trace, etc. have to be used, which are specificto different technologies and are mostly deployed by the development teams.However, in order to address specific requirements, it is preferable that teamscreate custom diagnostic solutions.Browser DiagnosticsInternet technologies running applications also face issues. Today, rich Internetapplications are available that have been built using AJAX, HTML5, FLEX,SilverLight, jQuery and similar technologies. These also present performancechallenges to Web developers. It is really difficult to identify bottlenecks on thebrowser that can impact the end-user experience. Diagnostic tools can help inthis process.Most of the client-side diagnostic tools are plug-ins to popular browsers. Thesecan be integrated with the browser and provide all the statistics related to pagerendering, Java script executions, memory, CPU utilization, etc. Some of thepopular diagnostic plug-ins includes Firebug for Mozilla, Speedtrace for Chrome,HttpWatch for IE.There are other set of tools, including Dynatrace Ajax Edition, that workindependently and extract the above information from the browser. These havemore features and provide dashboards and fine grained analysis for identifyingany existing issues.
Identifying Software Performance Bottlenecks using Diagnostic tools9Impetus’ Solution for Client Side DiagnosticsNot too many tools are available for mobile diagnostics, as multipletechnologies and platforms exist, and no single tool can serve the purpose. It isbetter to create custom diagnostic solutions that address specific requirementsof mobile applications.In few engagements, Impetus has used Aspect-oriented frameworks to build adiagnostic utility for mobile applications on the devices. The AOP frameworkslike AspectMe for J2ME and AspectLUA for LUA can be used for this purpose.These frameworks use byte code instrumentation to inject hooks into the targetapplication and measure runtime statistics, such as method timing and memoryutilization. A set of re-usable aspects can be created that can be hooked into theimportant method calls or UI rendering calls to track their performancestatistics.The aspects can be inserted using application binaries and hence there is noneed to undertake any code changes. The methods to be monitored arespecified at the time of instrumentation.In order to perform diagnostics for browsers to identify performancebottlenecks, we recommend installing proper plug-ins and configuring these tocollect performance statistics during browsing session.The plug-in gathers all the relevant details and makes them available to theperformance testing team. The team can then analyze the results to identifybottlenecks related to page loading, Java script execution, memory leaks, CPUutilization, etc.In one of our engagement by just tuning client side components, applicationperformance, especially the end user experience, was improved by almost 40%.
Identifying Software Performance Bottlenecks using Diagnostic tools10Benefits Offered by Diagnostics ToolsDiagnostic tools offer a range of benefits to performance testing anddevelopment teams within organizations, including the following:• Identification and isolation of performance bottlenecks, which may becomponents, methods, or SQL statements. This in turn, assists inresolving memory, exception, or other problems swiftly, thereby,reducing the efforts of the development team.• Complete application visibility across the lifecycle, ensuring higherquality when applications go live. They also help reduce the Mean Timeto Resolution in the application deployment environment.• Documentation of the root cause of any performance or stability issues,including the offending line of code, even in complex, highly distributedapplications. The performance testers do not need any programmingknowledge or access to source code to deliver an offline diagnosisreport that allows development to efficiently resolve issues.• Effective collaboration between performance engineers and thedevelopment team, due to the availability of detailed test results. Thisreduces the multiple iterations between both the teams and speeds upthe problem resolution time.• Improvement in the productivity of the Development team, includingdevelopers, architects and database engineers. The time-intensive,tedious, error prone and often impossible process of manuallycorrelating log files, from different back-end systems with the client-side load testing results, becomes obsolete. This significantlyaccelerates the problem resolution time.• Inbuilt intelligence and suggestions for teams based on best practices.Also, some of the tools can be directly integrated with the developer’senvironment and pinpoint the bottleneck at the code level itself.