2. IBM Rational Software Development Conference 2006
Session: SDP14
Agenda
IBM Performance Optimization Toolkit (IPOT) overview
IT Lifecycle Management
Problem Determination by Example
Profiling a J2EE Application
Resource Monitoring
Log Monitoring
Demo (30 min)
Online References
Q&As (10 min)
3. IBM Rational Software Development Conference 2006
Session: SDP14
IBM Performance Optimization Toolkit Overview
Who are the IPOT users?
Any developer who wants to identify root cause of
performance problems and accelerate problem determination
occurring in development environment.
Any tester who wants to identify root cause of performance
problems and accelerate problem determination occurring in
test environment.
Any support developer who wants to identify root cause of
performance problems and accelerate problem determination
occurring in production environment.
4. IBM Rational Software Development Conference 2006
Session: SDP14
IBM Performance Optimization Toolkit Overview
Why would they use IPOT?
IPOT provides transaction decomposition for application
optimization
Allows the developer/tester to monitor a distributed application in
real-time
Provides a Data Collection Infrastructure (DCI) for real-time
monitoring
Integrates with Rational Software Development Platform
IPOT accelerates problem determination by correlating the
different data collection views (logging, performance and
resource monitoring data) by time.
IPOT allows to import data from production environment and
visualize it for analysis and correlation in the development/test
environments.
5. IBM Rational Software Development Conference 2006
Session: SDP14
IT Lifecycle Management
ITBusiness
Application
Development
Problem
Determination
Development
Team
Deployment2
1
3
Operations
Team
6. IBM Rational Software Development Conference 2006
Session: SDP14
Problem Determination by Example
An example: Plants By WebSphere
Application ServerWeb Server Database Server
Application Server
Data Collection
Infrastructure
IPOT Agent
ITM Agent ITM Agent
Steps to collect resource data
Steps to collect application performance data
ITM Agent
7. IBM Rational Software Development Conference 2006
Session: SDP14
Instrumenting your Application Server
DCI provides a script to help instrument your application server
(instrumentServer.bat or instrumentServer.sh located in the
<INSTALL_HOME>rpa_prod directory).
Run without any parameters will display a help screen.
The WebSphere server that you want to instrument MUST BE
RUNNING before instrumentation and MUST BE RESTARTED
after instrumentation.
Example
instrumentServer –install –type IBM –serverName server1 –
serverHome “C:Program FilesWebSphereAppServer” –
serverVersion 5
BEA WebLogic™ is also supported
8. IBM Rational Software Development Conference 2006
Session: SDP14
Profiling a J2EE Application
Profile distributed applications in real-time
Import application trace data after an application has executed using
TMTP or ITCAM
20. IBM Rational Software Development Conference 2006
Session: SDP14
Architecture: Performance Data
Target SystemPresentation System
IPOT
RPT/RAD
Eclipse Platform
AgentController
AgentController
Data Collection
Infrastructure
Application Server
Application
IPOT Agent
21. IBM Rational Software Development Conference 2006
Session: SDP14
Resource Monitoring
Observe resource usage on systems as an application executes
Resource counters are statistical data that change over time
26. IBM Rational Software Development Conference 2006
Session: SDP14
Architecture: Statistical Data
Target System
IBM Tivoli Monitoring
Data Warehouse
Management Server
SOAPServer
Presentation System
IPOT
Web Service Client
RPT/RAD
Eclipse Platform
ITM Agent
(Linux OS)
ITM Agent
(Windows OS)
ITM Agent
(Unix OS)
ITM Agent
(DB2)
…
Internet
(HTTP/HTTPs)
27. IBM Rational Software Development Conference 2006
Session: SDP14
Steps to Problem Determination
Application log events are generated during execution
Correlation of these log events with events during resource
monitoring and application trace help identify the problem
Correlation between log events from different systems, such as
application servers and data base systems, help identify
problems
Common logging and trace format through Common Base
Event – OASIS standard
Over 115 adapters for log conversion (Generic Log Adapters)
33. IBM Rational Software Development Conference 2006
Session: SDP14
Log Monitoring
Common
Base Event
Application
Server
Servers
Storage
devices
Database
Networks
Applications
AdaptersAdapters
34. IBM Rational Software Development Conference 2006
Session: SDP14
Analyze applications deployed to a pre-production or
production environment using Tivoli products
Provides a developer a realistic view of the events in a
production environment for root cause analysis
Import performance data from IBM Tivoli Composite
Application Manager (ITCAM) products after the application
has executed
Import statistical data from IBM Tivoli Monitoring (ITM) after the
application has executed
Steps to Problem Determination
35. IBM Rational Software Development Conference 2006
Session: SDP14
Import Performance Data
36. IBM Rational Software Development Conference 2006
Session: SDP14
Import Performance Data
37. IBM Rational Software Development Conference 2006
Session: SDP14
Import Performance Data
38. IBM Rational Software Development Conference 2006
Session: SDP14
Import Performance Data
39. IBM Rational Software Development Conference 2006
Session: SDP14
Import Performance Data
40. IBM Rational Software Development Conference 2006
Session: SDP14
Import Performance Data
42. IBM Rational Software Development Conference 2006
Session: SDP14
Architecture: Import Performance Data
Target System
IBM Tivoli Composite Application
Manager
Data Warehouse
Management Server
WebServices
Presentation System
IPOT
Web Service Client
RPT/RAD
Eclipse Platform
Management
Agent
Management
Agent
Management
Agent
Management
Agent
…
Internet
(HTTP/HTTPs)
43. IBM Rational Software Development Conference 2006
Session: SDP14
Import Statistical Data
44. IBM Rational Software Development Conference 2006
Session: SDP14
Import Statistical Data
45. IBM Rational Software Development Conference 2006
Session: SDP14
Import Statistical Data
46. IBM Rational Software Development Conference 2006
Session: SDP14
Import Statistical Data
47. IBM Rational Software Development Conference 2006
Session: SDP14
Import Statistical Data
53. IBM Rational Software Development Conference 2006
Session: SDP14
Eric Labadie
Ashish Patel
http://www-128.ibm.com/developerworks/rational/library/05/523_perf/
Thank You
Editor's Notes
Please note that if you simply apply this template to your existing presentation, you risk not updating the notes and handouts masters of your presentation. Please follow the steps below to copy your existing slides into the new template.
1. Download the new template to your hardrive
2. Open existing presentation that needs to be updated with the new template
3. Go to slide sorter view from the &quot;View&quot; menu
4. Press &lt;Control&gt; &quot;A&quot; to select all slides in this view
5. Press &lt;Control&gt; &quot;C&quot; to copy all slides in this view
6. Select &quot;Open&quot; from the File menu and open the new template
7. Go to slide sorter view from the &quot;View&quot; menu
8. Press &lt;Control&gt; &quot;V&quot; to paste the slides
9. Select slides 1 and 2 which are no longer needed in your presentation and press &lt;Control&gt; &quot;X&quot; to cut
10. Select “Save As” from the File menu and rename the file
Reformatting Issues:
Replace Fonts (especially Times New Roman)
Due to a PowerPoint limitation, it’s advisable to run the “replace fonts” option after applying the template
Select Format/Replace Fonts and select desired font to be changed (Times New Roman and Arial are commonly seen here to be replaced with Arial Narrow)
Correct Colors - graphic objects and text created not using the auto layout features may not automatically convert. Some reformatting may be necessary.
If slide background colors are incorrect, reset slide background color to autocolor
If graphics are colored incorrectly, use the colors that are built-in to the template already as these are pre-approved colors
Slide Layouts
It may be necessary to reapply slide layouts to problem slides. To do this, from the View menu, make sure “Task Pane” is selected. Select “Slide Layout” from the Task Pane and select the desired layout to reapply. Sometimes this action needs to be applied twice in order for layout to readjust.
i/t has to be on demand just as much as the business has to be on demand w/ their clients =&gt; IBM has to respond to that
- Optimize application before deploying to prevent problems before it happens in the first place
Reduce business downtimewhile accelerating business value throughput…
Quickly discover and understand application-level errors even after deployment
Speed Tivoli-aware application fix and (re)build
Optimize and accelerate (re)deployment
…by bridging developmentand operations teams
Example: Trying to run a test to identify if performance problems exist in a multi-user environment.
Diagram: Can be distributed on separate machines.
Workbench
collect performance and resource monitoring data
import data from historical systems
Data Collection Infrastructure (DCI)
Data Collection Agent
IBM Tivoli Data Collection
IBM Remote Agent Controller
IPOT Agent is reported the ARM Events. ARM Events are collected and organized into a transactional hierarchy, from the root transaction to all of its sub-transactions. This hierarchy is then converted into TPTP Trace events and sent to the Presentation System.
Need DCI installed on the Presentation System if trying to Profile J2EE Performance Metrics while executing a Performance Schedule or Test.
Resource Monitoring
We provide data collection (via the RAC) from Windows/Linux machine, JBoss/JOnas
IPOT adds value by having the ITM infrastructure in place because they support a wide array of ITM agents
There are two WebSphere configuration files that get changed during instrumentation:
(WAS5) &lt;WAS_HOME&gt;\config\cells\&lt;cell_name&gt;\nodes\&lt;node_name&gt;\servers\&lt;server_name&gt;\server.xml and variables.xml.
(WAS 6) &lt;WAS_HOME&gt;\profiles\&lt;profile_name&gt;\config\cells\&lt;cell_name&gt;\nodes\&lt;node_name&gt;\servers\&lt;server_name&gt;\server.xml and variables.xml.
To manually verify that instrumentation was successful, check that these two files have changed from their pre-instrumentation state.
In particular, check in server.xml that the ‘jvmEntries’ element has a new string in the ‘genericJvmArguments’ attribute.
Websphere Express is NOT supported. Only Websphere Application Server is supported.
- genericJvmArguments=“-Xbootclasspath/a:${MA_INSTRUMENT}\lib\jiti.jar;${MA_INSTRUMENT}\lib\bootic.jar;${MA_INSTRUMENT}\ic\config;${MA_INSTRUMENT_APPSERVER_CONFIG} -Dma.instrument=${MA_INSTRUMENT} -Dma.appserverconfig=${MA_INSTRUMENT_APPSERVER_CONFIG} -Dtmtp.user.dir=C:\Programs\IBM\Rational\SDP\6.0\rpa_prod\TIVOLI~2 -Dcom.ibm.tivoli.jiti.config=${MA_INSTRUMENT_APPSERVER_CONFIG}\config.properties -Dcom.ibm.tivoli.transperf.logging.qualDir=${MA_LOG_QUALDIR} -Dcom.ibm.tivoli.jiti.probe.directory=C:\Programs\IBM\Rational\SDP\6.0\rpa_prod\TIVOLI~2\app\instrument\5301\lib\ext -Dws.ext.dirs=C:\Programs\IBM\Rational\SDP\6.0\rpa_prod\TIVOLI~2\app\instrument\5301\lib\ext -Djlog.propertyFileDir=${MA_INSTRUMENT_APPSERVER_CONFIG} -Xrunvirt_5301:agent=ijitipi_5301:${MA_INSTRUMENT_APPSERVER_CONFIG}\jiti.properties,agent=piAgent:server=enabled
You will see a number of additional entries in variables.xml with the attribute symbolicName prefixed with “MA”.
E.g.
&lt;entries xmi:id=&quot;VariableSubstitutionEntry_1115417429895&quot; symbolicName=&quot;MA_LOG_QUALDIR&quot; value=&quot;J2EE/server1_100&quot; description=&quot;Base directory for logging for the TMTP Management Agent&quot;/&gt;
There is also an IPOT DCI specific file that gets updated when instrumentation occurs.
This file is located in &lt;INSTALL_HOME&gt;\rpa_prod\tivoli_comp\app\instrument\5301\bin\InstrumentationRegistry.xml
This file contains an entry for every server that has been successfully instrumented. The entry will look something like:
&lt;applicationServer type=&quot;IBM&quot;&gt;
&lt;name&gt;server1&lt;/name&gt;
&lt;path&gt;C:\WebSphere\AppServer6&lt;/path&gt;
&lt;type&gt;2&lt;/type&gt;
&lt;secured&gt;false&lt;/secured&gt;
&lt;keepPiAgentStr&gt;false&lt;/keepPiAgentStr&gt;
&lt;/applicationServer&gt;
Circled launch configuration types are IPOT-specific. Others come from TPTP, RPT, RAD and Eclipse.
ARM Instrumented Application is for profiling non-J2EE applications or custom ARM instrumented applications.
ARM – Application Response Measurement, an open standard.
Use J2EE Application when profiling a J2EE Application that is load driven from a browser.
IPOT contributes two profiling sets
J2EE Performance Analysis
ARM Performance Analysis
Profiling sets are configurations of profiling types with specific options, much like launch configurations. Users can define custom sets.
Profiling types have options to indicate what to include and exclude during data collection
These two sets apply only to the IPOT agent, not for the TPTP JVMPI (Java Virtual Machine Profiler Interface) agent.
JVMPI agent resides on the same process as application being profiled
IPOT agent behaves similar to a service, as it runs in its own process, and profiles all ARM instrumented applications that execute on the system
IPOT agents support a distributed environment and the workbench will automatically display all IPOT agents involved in a distributed transaction.
Components tab in J2EE Performance Analysis lets the user specify which J2EE components they wish to collect data from.
This is applicable for both IBM Websphere™ and BEA WebLogic ™ application servers.
We can also profile applications that have been instrumented with custom ARM calls.
ARM profiling type allows you to filter applications and transactions to profile, and allows you to do sampling of transactions.
Application filter matches the name of the application that was registered with the ARM engine. All ARM-enabled apps must set this.
Transaction filter matches the ARM transaction name. This is an arbitrary name set by the ARM-enabled application.
Sampling tab allows you to control the frequency of which data is collected.
This tab is the same for ARM and J2EE Performance Analysis
IPOT contributes two profiling sets
J2EE Performance Analysis
ARM Performance Analysis
Profiling sets are configurations of profiling types with specific options, much like launch configurations. Users can define custom sets.
Profiling types have options to indicate what to include and exclude during data collection
These two sets apply only to the IPOT agent, not for the TPTP JVMPI (Java Virtual Machine Profiler Interface) agent.
JVMPI agent resides on the same process as application being profiled
IPOT agent behaves similar to a service, as it runs in its own process, and profiles all ARM instrumented applications that execute on the system
IPOT agents support a distributed environment and the workbench will automatically display all IPOT agents involved in a distributed transaction.
End to end transaction UML view with a performance problem for the problematic transaction. On the scale on the right, the user can see the different shades of red beside each method calls. The darker red on the scale, the more time is spent in this location in the problematic transaction. The user can then jump to the source code from this view.
The user can the switch to the Method details views to view as well the time spent in the descendant methods called from this transaction. The user can then jump to the source code from this view.
As well, we provide method statistics views showing to the user how many times a methods was called and what is the average time spent in this method. The user can then jump to the source code from this view.
Finally, the user can go to source code from the previous views to identify performance problem
Workbench
collect performance and resource monitoring data
import data from historical systems
Data Collection Infrastructure (DCI)
Data Collection Agent
IBM Tivoli Data Collection
IBM Remote Agent Controller
Workflow:
The RPT client is the edge of the transaction (where ARM transactions are first generated). Therefore, all page are transacted from the presentation system, which behaves similar to a browser, by placing HTTP requests for all page elements (belonging to their respective pages).
For all HTTP Requests, RPT adds the ARM_CORRELATOR header attribute to the request.
Multiple RPT clients can generate the same load for the same transactions and they will be collected by the ARM engine independently.
Anything downstream from the RPT client (for example a webserver or J2EE appserver) must be instrumented with J2EE Monitoring Component (or Tivoli Data Collection, which consists of probes or hooks) that knows how to detect the ARM_CORRELATOR header attribute and then make the appropriate ARM calls to the ARM engine. Once the probe makes the ARM call, the transactions are all treated the same by the ARM engine.
In order to see “into” the application (at the method level) when the RPT test/schedule is executed, the Execution Environments involved muse be instrumented, just the same as one would do with TMTP or the IPOT DCI, so that the RPT HTTP Requests can be correlated with the AppServer’s behavious.
Caspian: Architecture will change so that IPOT is not dependent on RPT, however, RPT becomes dependent on IPOT. Allowing other products to leverage the toolkit.
IPOT Agent is reported the ARM Events. ARM Events are collected and organized into a transactional hierarchy, from the root transaction to all of its sub-transactions. This hierarchy is then converted into TPTP Trace events and sent to the Presentation System.
Need DCI installed on the Presentation System if trying to Profile J2EE Performance Metrics while executing a Performance Schedule or Test.
Circled launch configuration types are either IPOT-specific or come from TPTP.
Tivoli Monitored Host is for monitoring counters from IBM Tivoli Monitoring (ITM)
Host – Linux monitors Linux-based operating systems via the Rational Agent Controller (RAC)
Host – Windows is perfmon monitoring via the RAC
J2EE App Server - JBoss is monitoring via JMX
J2EE App Server – JOnAS is monitoring via JMX
The user can also import different log files from production as well into an aggregated view with filtering and analysis results using the symptom database
Steps to determine the cause of the problem:
using performance data
Collect performance data using the DCI
Performance data allows the tester and developer to peer into the behaviour of the application or service at a programmatic level (ie. The method level)
This approach provides solid evidence of where the potential problem is located and increases efficiency in problem determination between the tester and developer, thereby, effectively reducing the time to identify and diagnose performance problems.
Real-time data collection using the Data Collection Infrastructure (DCI) for ARM-instrumented applications
Application Response Measurement (ARM) is an open standard from OpenGroup
End to end transaction UML view with a performance problem for the problematic transaction. On the scale on the right, the user can see the different shades of red beside each method calls. The darker red on the scale, the more time is spent in this location in the problematic transaction. The user can then jump to the source code from this view.