This document discusses an application performance monitoring tool called AppInternals Xpert. It provides three key functions: (1) code-level transaction tracing across tiers, (2) collection of high-resolution performance metrics across all application components, and (3) memory analysis to detect leaks. The tool offers real-time and historical dashboards and scales to monitor hundreds of servers with low overhead through automatic discovery and auto-tuning.

1. Monitoreo y Análisis de
aplicaciones “Multi-Tier”
 Eduardo Lopes
 elopes@opnet.com
#GX3009

2. Objectives
 Provide an introduction to core concepts including
 Cross-tier code-level Transaction Tracing
 Application and system-level Performance Metrics
 Memory Analysis
2 CONFIDENTIAL – RESTRICTED ACCESS: This information may not be disclosed, copied, or transmitted in any format without OPNET’s prior written consent.
© 2012 OPNET Technologies, Inc. OPNET Technologies Inc., OPNET, OPNETWORK, and all OPNET hardware and software product names are trademarks of
OPNET. All other trademarks are the property of their respective owners.

3. Introduction to AppInternals Xpert - What is
AppInternals Xpert?
 Broad and deep coverage of all application
components
• Cross-tier, code-level transaction tracing of all Java
and .NET transactions
• High-resolution (1 second) performance metrics
collected across all tiers including Java, .NET,
database, Web Server, Operating System, VMware
OPNET Dashboards
 Real-Time and Historical dashboards
 Three distinct analysis engines
• Transaction Tracing
• Statistical metrics correlation
• Memory analysis
 Easy setup and maintenance
• Automatic application discovery Metric Insights
• Dynamically-established thresholds
• Intelligent auto-tuning to ensure low overhead
 Designed for Production, supports QA workflows
 Scalable architecture supports hundreds of servers
Transaction Tracing
3 CONFIDENTIAL – RESTRICTED ACCESS: This information may not be disclosed, copied, or transmitted in any format without OPNET’s prior written consent.
© 2012 OPNET Technologies, Inc. OPNET Technologies Inc., OPNET, OPNETWORK, and all OPNET hardware and software product names are trademarks of
OPNET. All other trademarks are the property of their respective owners.

4. Deep Application Component Monitoring
 Most comprehensive performance management solution for web
applications
 Broad and deep coverage of all application components
• Thousands of performance metrics collected across all tiers Memory Leak Detection
incl. Java, .NET, database, VMware, and SOA
• High resolution data collection (~1 second)
 Real-time and historical dashboards
 Three distinct analysis engines:
• Statistical metrics correlation
• Transaction tracing
• Memory analysis Top Offenders
 Easy setup and maintenance
Time/Statistical Correlation
• Automatic application discovery
• Dynamically-established thresholds
• Intelligent auto-tuning to ensure low overhead
 Designed for production, supports QA workflows
 Scalable architecture supports hundreds of servers
True Statistical Correlation
4 CONFIDENTIAL – RESTRICTED ACCESS: This information may not be disclosed, copied, or transmitted in any format without OPNET’s prior written consent.
© 2012 OPNET Technologies, Inc. OPNET Technologies Inc., OPNET, OPNETWORK, and all OPNET hardware and software product names are trademarks of
OPNET. All other trademarks are the property of their respective owners.

5. Deep Visibility for Java and .NET
OPNET Confidential – Not for release to third parties
5 © 2010 OPNET Technologies, Inc. All rights reserved.ACCESS: This information may notnames are trademarks of OPNET any format without OPNET’s prior written consent.
CONFIDENTIAL – RESTRICTED
OPNET and OPNET product be disclosed, copied, or transmitted in Technologies, Inc.
© 2012 OPNET Technologies, Inc. OPNET Technologies Inc., OPNET, OPNETWORK, and all OPNET hardware and software product names are trademarks of
5 OPNET. All other trademarks are the property of their respective owners.

6. Three Complementary Engines
 Code-level Transaction Tracing
• Reveals exactly what happened in
a particular transaction
 Where in the code is time spent?
 What needs to be fixed/optimized?
• Identifies dependencies on code/libraries/external systems Code-level transaction tracing
 Metric Correlation Engine
• Identifies unexpected relationships among components across
many tiers
• Does not require developer/code-specific knowledge to interpret
• Explicitly identifies when systems are behaving abnormally
• Visibility into non-Java/.NET bottlenecks: within the
OS, Database, etc.
Metric correlation engine
 Memory Analysis
• Identifies the actual sources of memory leaks, not just that
they exist
• Addresses prevalent problem: memory leaks are notoriously
difficult to track down
Memory analysis
6 CONFIDENTIAL – RESTRICTED ACCESS: This information may not be disclosed, copied, or transmitted in any format without OPNET’s prior written consent.
© 2012 OPNET Technologies, Inc. OPNET Technologies Inc., OPNET, OPNETWORK, and all OPNET hardware and software product names are trademarks of
OPNET. All other trademarks are the property of their respective owners.

7. „Always-On‟ Transaction Tracing
 Code-level visibility into individual user transactions
• Trace user transactions through the code: from URL to SQL
• Both .NET and Java
 Low overhead, suitable for production environments
 Forensic events: Automatic archiving of problematic transactions
Detailed call stack with code-level visibility
7 CONFIDENTIAL – RESTRICTED ACCESS: This information may not be disclosed, copied, or transmitted in any format without OPNET’s prior written consent.
© 2012 OPNET Technologies, Inc. OPNET Technologies Inc., OPNET, OPNETWORK, and all OPNET hardware and software product names are trademarks of
OPNET. All other trademarks are the property of their respective owners.

8. Multi-Tier Forensic Captures
 Enables analysis of thousands of saved traces across multiple Java and .NET tiers
 Captures are triggered automatically based on user-defined policies
 Leverages powerful transaction analysis environment
8 CONFIDENTIAL – RESTRICTED ACCESS: This information may not be disclosed, copied, or transmitted in any format without OPNET’s prior written consent.
© 2012 OPNET Technologies, Inc. OPNET Technologies Inc., OPNET, OPNETWORK, and all OPNET hardware and software product names are trademarks of
OPNET. All other trademarks are the property of their respective owners.

9. Always Low Overhead: Auto-Tuning
 Auto-tuning reduces blind spots by enabling more Java/.NET packages to be
instrumented overall
• AppInternals Xpert‟s unique auto-tuning scans the code to instrument „interesting‟ methods
only (e.g. larger methods, methods with loops, methods that call other classes)
• Certain methods exponentially increase the total overhead while providing no valuable
information for troubleshooting – these “hotspots” are additionally identified and removed at
runtime
• Performance data is recorded only for methods that represent a percentage (configurable) of
overall application activity
Manual Instrumentation AppInternals Xpert Auto-tuning
Frequently called methods of short duration
More interesting methods
9 CONFIDENTIAL – RESTRICTED ACCESS: This information may not be disclosed, copied, or transmitted in any format without OPNET’s prior written consent.
© 2012 OPNET Technologies, Inc. OPNET Technologies Inc., OPNET, OPNETWORK, and all OPNET hardware and software product names are trademarks of
OPNET. All other trademarks are the property of their respective owners.

10. Application and System-Level Performance Metrics
Collects metrics across a wide range of components
 Application Servers  Web Servers
•
Apache Tomcat • Apache
•
ATG Dynamo • IIS
•
Caucho Resin  Operating systems
•
JBoss • AIX
•
Macromedia ColdFusion • HP-UX
•
Macromedia JRun • Linux (Red Hat, SUSE, zLinux)
•
Microsoft .NET-based Applications • Solaris
•
Mort Bay Jetty • Windows
•
Oracle Application Server
Sun Java System Application Server  General purpose
•
•
WebLogic • SNMP
•
WebSphere • VMware Guest
• WMI
 Databases • Generic Scripting
• DB2 • SDK
• Oracle • VMware DataCenter
• SQL Server • Storage Systems
• Sybase • Hadoop
10 CONFIDENTIAL – RESTRICTED ACCESS: This information may not be disclosed, copied, or transmitted in any format without OPNET’s prior written consent.
© 2012 OPNET Technologies, Inc. OPNET Technologies Inc., OPNET, OPNETWORK, and all OPNET hardware and software product names are trademarks of
OPNET. All other trademarks are the property of their respective owners.

11. Use Cases
 QA/Testing
• Problem: A load test indicates unacceptable response time when the
load increases from 40 to 50 users, but what is causing the
degradation?
• Solution: OPNET AppInternals Xpert‟s advanced analytics compares
the 40 vs. 50 user scenarios, shows potential sources of poor
performance with drill downs into specific application components and
transactions.
 Production Monitoring
• Problem: The application is slow but the network is fine and the
servers are up, so where is the problem?
• Solution: OPNET AppInternals Xpert quickly shows the slowest
running transactions over a period of time as well as sifts through
thousands of metrics in the application, across all servers, and helps to
rapidly isolate the poorly performing components -- dramatically
reducing the mean time to repair (MTTR)
11 CONFIDENTIAL – RESTRICTED ACCESS: This information may not be disclosed, copied, or transmitted in any format without OPNET’s prior written consent.
© 2012 OPNET Technologies, Inc. OPNET Technologies Inc., OPNET, OPNETWORK, and all OPNET hardware and software product names are trademarks of
OPNET. All other trademarks are the property of their respective owners.

12. Questions that AppInternals Xpert Can Answer
Examples
 What was a particular user doing during a period of time?
 How many times did a particular transaction occur during a period of time?
 Are the load balancers doing their job?
 Which database instances are most transactions being executed on?
 Which transactions are causing exceptions?
 Are there any servers from one environment making calls to other
environments?
 Is there a relationship between CPU spikes and execution of particular
EJBs, Servlets or .NET Managed Components?
 Which components of my application perform poorly with unusually heavy
workloads?
 As load on an application increases, at what point does throughput
plateau?
 How is memory being allocated and released on my JVM/CLR and how is
this effecting performance?
12 CONFIDENTIAL – RESTRICTED ACCESS: This information may not be disclosed, copied, or transmitted in any format without OPNET’s prior written consent.
© 2012 OPNET Technologies, Inc. OPNET Technologies Inc., OPNET, OPNETWORK, and all OPNET hardware and software product names are trademarks of
OPNET. All other trademarks are the property of their respective owners.

13. Performance Metrics
What AppInternals Xpert Collects
 Within Java and .NET applications
• Code-level including load, throughput, response times, exceptions
• Down to method level for component types (ADO, JDBC, Servlet/JSP,
ASP..)
• Memory (objects allocated, free heap, consumed, garbage
collections..)
 Infrastructure
• Operating systems - Windows, Linux (Red Hat, SUSE), AIX, HP-UX,
Solaris
• VMware, Storage Systems, WMI and SNMP
 Application Tiers
 Components running on front-end, back-end, and middleware
 Databases, Web Servers, App Servers
 Extendable through Scripting, SDK and share.opnet.com
• REST API, scripting API, C/C++ API
13 CONFIDENTIAL – RESTRICTED ACCESS: This information may not be disclosed, copied, or transmitted in any format without OPNET’s prior written consent.
© 2012 OPNET Technologies, Inc. OPNET Technologies Inc., OPNET, OPNETWORK, and all OPNET hardware and software product names are trademarks of
OPNET. All other trademarks are the property of their respective owners.

14. Performance Metrics
Dynamic Baselines, Deviation and Correlation Analysis
 Dynamic Baselines
• Automatically determines “normal” range for every metric
“Normal”
• Production vs. QA/Test Mode
 What is considered “normal” can be very different
 Calibrate the analysis engine with a reference load
 Deviations
• Show abnormalities in observed metric values
relative to dynamic baselines
“Abnormal”
• Deviation score is calculated across large numbers
(thousands) of metrics to determine what is the
“most abnormal”
 Correlations
• Take a metric of interest and see other metrics
across entire application stack whose behavior is
correlated to that metric
• Finds relationships among components, unknown “Correlation”
cause effect, root cause
14 CONFIDENTIAL – RESTRICTED ACCESS: This information may not be disclosed, copied, or transmitted in any format without OPNET’s prior written consent.
© 2012 OPNET Technologies, Inc. OPNET Technologies Inc., OPNET, OPNETWORK, and all OPNET hardware and software product names are trademarks of
OPNET. All other trademarks are the property of their respective owners.

15. Managed Memory Concepts
 Symptoms of a Memory Leak
• Out of memory exceptions.
• CPU utilization goes up because GCs are taking longer.
• Garbage collection runs more frequently, as the application attempts to
free up space in an ever decreasing pool of usable memory. Thus
causing the application to slow down.
15 CONFIDENTIAL – RESTRICTED ACCESS: This information may not be disclosed, copied, or transmitted in any format without OPNET’s prior written consent.
© 2012 OPNET Technologies, Inc. OPNET Technologies Inc., OPNET, OPNETWORK, and all OPNET hardware and software product names are trademarks of
OPNET. All other trademarks are the property of their respective owners.

16. Memory Analysis
Object Sampling Metrics
 AppInternals Xpert has the ability to provide live object counts and the
allocation rates of different objects
 Requires the “memanalysis” setting in Java and .NET instrumentation adapters
and Java version 1.5 or later
Classes with the highest live object count and allocation rates
16 CONFIDENTIAL – RESTRICTED ACCESS: This information may not be disclosed, copied, or transmitted in any format without OPNET’s prior written consent.
© 2012 OPNET Technologies, Inc. OPNET Technologies Inc., OPNET, OPNETWORK, and all OPNET hardware and software product names are trademarks of
OPNET. All other trademarks are the property of their respective owners.

17. Memory Analysis
Heap Snapshots
 Records detailed information about
memory usage in Java and .NET
applications at a particular time
 Use to identify the cause of complex
heap problems, when the heap analysis
metrics are not enough
 Can identify specific method and source
code line number responsible for
memory leaks or excessive heap
throughput
 Depending on the size of the
application‟s heap, this can be resource
intensive and suspend the application
for some time
 Run in Dev/QA/Pre-production
 Use with load balancer – remove node
from cluster, when all users session have
ended then run snapshot
17 CONFIDENTIAL – RESTRICTED ACCESS: This information may not be disclosed, copied, or transmitted in any format without OPNET’s prior written consent.
© 2012 OPNET Technologies, Inc. OPNET Technologies Inc., OPNET, OPNETWORK, and all OPNET hardware and software product names are trademarks of
OPNET. All other trademarks are the property of their respective owners.

18. Conferencias que brindaremos
 Incorporación de procesos de Ingenieria de Performance dentro de
su organización
• Hoy 17:15
 Presentación de las soluciones de OPNET para la Gestión de
Performance de Aplicaciones (APM)
• Mañana 9:30
 AppResponse Xpert SaaS Edition
• Mañana 11:45
18 CONFIDENTIAL – RESTRICTED ACCESS: This information may not be disclosed, copied, or transmitted in any format without OPNET’s prior written consent.
© 2012 OPNET Technologies, Inc. OPNET Technologies Inc., OPNET, OPNETWORK, and all OPNET hardware and software product names are trademarks of
OPNET. All other trademarks are the property of their respective owners.

19. GRACIAS!!