Websphere doctor - your guide to diagnose issues

WebSphere Doctor
By Joseph Amrith Raj
A series of articles covering different troubleshooting tools and techniques for WebSphere Application server

Index of tools
Chapter1: Health Center
Chapter2: Class loader Viewer
Chapter3: Thread Dump and Monitor Analyzer [TDMA]
Chapter4: Garbage Collection and Memory Visualizer
Chapter5: Visual Configuration Explorer [VCE]
Chapter6: Integrated Tivoli performance viewer [TPV]
Chapter7: Recover lost Key data base password
Chapter8: IBM Support assistant v5
Chapter9: Web Server issues
Chapter10: Plug-in issues
Chapter11: Application Server logs [will be added in version2]
Chapter12: Tracing [will be added in version2]

Joseph Amrith Raj
http://facebook.com/webspherelibrary
Tool - 1: Health Center

2 Tool - 1: Health Center
Joseph’s WebSphere Library
Table of Contents
WebSphere Application Server Troubleshooting.......................Error! Bookmark not defined.
About Health Center ................................................................................................................... 3
Installing and configuring HC ........................................................................................................ 5
Step-1 : HC tool in ISA ................................................................................................................ 5
Step -2 : HC Agent....................................................................................................................... 6
Step-3 : HC configuration for WebSphere Application Server ....................................................7
Starting and using HC..................................................................................................................... 8
Viewing and analyzing the collected data......................................................................................12
Classes........................................................................................................................................12
Environment ..............................................................................................................................12
Garbage collection......................................................................................................................13
I/O..............................................................................................................................................14
Locking.......................................................................................................................................15
Native Memory...........................................................................................................................16
Profiling......................................................................................................................................17
References......................................................................................................................................19
About Author .................................................................................................................................19
Connect With US........................................................................................................................... 20

In the previous part of this series [here], we discussed how to install the IBM support assistant
workbench [ISA]. ISA is complementary software offering which provides you with a workbench
to help you with problem determination. With a focus on quickly finding key information,
automating repetitive steps and arming you with a variety of serviceability tools, you’ll be
prepared for self-analysis and diagnosis of problems and faster time to resolution.
About Health Center
In this, we will discuss one of the diagnose tool available in ISA to analyze the problems. The
tool is Health Center. Health Center is a very low overhead monitoring tool. It runs alongside an
IBM Java application with a very small impact on the application's performance. Health Center
monitors several application areas, using the information to provide recommendations and
analysis that help you improve the performance and efficiency of your application. Health
Center can save the data obtained from monitoring an application and load it again for analysis
at a later date.
Health Center provides visibility, monitoring and profiling in the following application areas:
Performance
 Java method profiling: The Health Center uses a sampling method profiler to
diagnose applications showing high CPU usage. It's low overhead which means
there is no need to specify in advance which parts of the application to monitor;
the Health Center simply monitors everything. It works without recompilation or
byte code instrumentation and shows where the application is spending its time,
by giving full call stack information for all sampled methods.
 Lock analysis: Synchronization can be a big performance bottleneck on multi-
CPU systems. It is often difficult to identify a hot lock or assess the impact
locking is having on your application. Health Center records all locking activity
and identifies the objects with most contention. Health Center analyses this
information, and uses it to provide guidance about whether synchronization is
impacting performance
 Garbage collection: The performance of Garbage Collection (GC) affects the
entire application. Tuning GC correctly can potentially deliver significant
performance gains. Health Center identifies where garbage collection is causing
performance problems and suggests more appropriate command line options.
Memory usage
The Health Center will identify if your application is using more memory than seems
reasonable, or where memory leaks occur. It then suggests solutions to memory issues, as
well as Java heap sizing guidance.
System Environment

Health Center uses an 'environment perspective' to provide details of the Java version,
Java classpath, boot classpath, environment variables, and system properties. This is
particularly useful for identifying problems on remote systems or systems where you do
not control the configuration. If the Health Center detects mis-configured applications, it
will provide recommendations on how to fix it.
Java Class loading
Health Center provides class loading information, showing exactly when a class has been
loaded and whether it is cached or not. This helps you determine whether your
application is being affected by excessive class loading.
File input and output
Health Center uses an 'I/O perspective' to monitor application input or output (I/O) tasks as
they run. You can use this perspective to monitor how many and which files are open,
and to help solve problems such as when the application fails to close files.
Real Time Monitoring
The WebSphere Real Time perspective (WRTP) helps you identify unusual or exceptional
events that might occur when you run a WRT application. The trace information can be
presented in various ways, including linear or logarithmic scales, and histograms. WRTP
provides some pre-defined trace point views that are especially helpful.
Object Allocations
Health Center allows you to view the size, time and code location of an object allocation
request that meets specific threshold criteria.
Health Center [HC] is a client-server model tool. This means, you can run the server anywhere
and then you install a client into your JVM and connect it to the server. For example, I’ve the
server side component installed on my local machine and the clients are installed to all the
development websphere application server machine. Now I can connect to the server on my
local machine and monitor any server.

Installing and configuring HC
Step-1 : HC tool in ISA
 Open the ISA tool
 Go to update Find new 
 Here you can find product and tool add-ons .
 Go to tool add-ons
 Expand the JVM based tools section and select, IBM Monitoring and Diagnostic tools for
Java – Health Center

 Click Next, accept the license terms and install. You need to restart ISA before you can
use HC.
Step -2 : HC Agent
With the above steps, the configurations on the server side component of the HC are complete.
The next step is to configure the client/JVMs.
Download the required HC agent for your platform form here
Extract the downloaded zip file. You’ll see
Now copy these 3 files to the appropriate directories of websphere JAVA. For Example, if your
websphere application server java is installed in /opt/ibm/websphere/java.. then copy:

1. healthcenter.so/dll to /opt/ibm/websphere/java/jre/bin
2. healthcenter.properties to /opt/ibm/websphere/java/jre/lib
3. healthcenter.jar to /opt/ibm/websphere/java/jre/lib/ext
Step-3 : HC configuration for WebSphere Application Server
The next step is to configure websphere application server.
 First, check what version of the JAVA you are using
 In the administration console, go to servers server type websphere application
servers server_name java and process management process definition java
virtual machine generic JVM arguments and the following:
On UNIX/Linux systems:
 For Java 5 SR9 and earlier or Java 6 SR4 and earlier, use:
-agentlib:healthcenter -Xtrace:output=/tmp/perfmon.%p.out
 For Java 5 SR10 and later, or Java 6 SR5 and later, use:
-Xhealthcenter
On Windows:
 For Java 5 SR9 and earlier or Java 6 SR4 and earlier, use:
-agentlib:healthcenter -Xtrace:output=C:tempperfmon.%p.out
 For Java 5 SR10 and later, or Java 6 SR5 and later, use:
-Xhealthcenter
 Apply the change and restart the JVM.
The setup is complete. The default port used by HC agent is 1972. However you can change this
in JAVA_HOME/jre/lib/healthcenter.properties

Starting and using HC
HC is a tool available in ISA , means first we need to launch and then launch the HC tool. After
launching the ISA, click on analyze the issues.

Now you need enable an application for monitoring.

The red box in above screenshot is for the details of the HC agent. The green box is for
authentication details to connect to the HC agent.
After you enter the details and click Next, it will display the JVMs which can be monitored.
Select a JVM and click Finish.

The above screenshot is with all the default configurations. However you can customize the
amount of data collected using ‘Monitored JVM data collection settings’

Viewing and analyzing the collected data
There are 7 perspectives available in the HC to view the collected data.
Classes
This prespective will display the class loading over the time. The starting point is when the JVM
is started. It also shows weather the class is loaded from the shared cache.
Usage: Will be helpful to analyze the out of memory errors. When a class is loaded, it will not be
released until the classloader which loaded it is garbage collected. The loaded classes will occupy
the heap memory. So, if more classes are loaded than the classes the unloaded, it might lead to
outofmemory errors.
Environment
This perspective shows system and configuration information about the monitored JVM. Also,
Analysis and recommendations panel will help you understand, if any parameters can affect
system performance, stability and serviceability. [red box in below screenshot]

Garbage collection
This perspective will show, heap usage graph, pause times graph and shows recommendations.
Used heap memory graph shows the memory usage by the applications on that JVM. It shows
the heap size and heap size after garbage collection.
Pause times chart shows the affect of garbage collection [GC] cycles. When GC is running, all the
application threads are stopped. Longer garbage collection pauses are often associated with
better application throughput and are not a performance problem. Spending extra time in
garbage collection can often lead to improved memory allocation and memory access times. The
aim of garbage collection tuning is to have reduced pause times only if low response times are
required.

Object allocation tab [blue box in above screenshot] will let you identify which code is allocating large
objects. The table shows, the size of the object allocated, time of the allocation and the code
which allocated the request.
Usage: helps you identify memory leaks and tune the heap sizes.
I/O
This perspective provides information about I/O activities performed by the JVM, like file open
events, file close events and details files that are currently open.

Locking
This perspective provides information in graph and table form that helps you understand any
contention caused by locking.

Native Memory
Native memory is the memory provided to the Java process by the operating system. The
memory is used for heap storage and other purposes. This perspective provides information
about the native memory usage of the process and system.

Profiling
This perspective helps you understand the work performed by the applications. You can use
method profiling to see the methods that consume the most resources.

Method profile view [red box in screenshot], shows which methods are using most processing resource.
 Invocation paths [green box in screenshot], shows the method that called the highlighted method.
 Called methods [blue box in screenshot], shows where the highlighted method is doing work.
 Timeline [brown box in screenshot], shows when the methods are invoked.

References
 Overview of health center - http://www.youtube.com/watch?v=5Tcktcl0qxs
 Install ISA and HC - http://www.youtube.com/watch?v=6WjE9U0jvEk
 How to enable a java app for HC -
http://www.youtube.com/watch?v=Hdp0mJ13NLQ
 About ISA :
http://websphereapplicationservernotes.wordpress.com/2012/03/15/websphere
-application-server-troubleshooting-part-0-ibm-support-assistant/
About Author
Joseph Amrith Raj is a technical consultant for multiple WebSphere products. He worked on
various product consulting and support teams including WebSphere Application Server,
WebSphere MQ, WebSphere Message Broker, Enterprise Service Bus and WebSphere Process
Server. He has 7 years of experience in administration, troubleshooting, consulting and he has
significant experience in architecture, strategy and leadership positions. He is IBM certified for
WAS, WMQ, WPS , SOA and Cloud Computing.

Connect With US
 Get notified on upcoming tutorials
 Subscribe, to get this WebSphere blog posts directly to your inbox : here
 Tell us your feedback and what you want
 Please leave your comments in the comments section below the post
 Send us suggestions for future topics and improvements about the content at
webspherelibrary@live.com
 Be Connected!
 Connect with us on Facebook
 Connect with us on Twitter
 More about us : profile

Joseph Amrith Raj
Tool -2: Class loader viewer

Joseph’s WebSphere Library Page 2
Contents
What does a class loader do? .......................................................................................................................3
Class not found issue ....................................................................................................................................3
Troubleshooting............................................................................................................................................3
Class loader viewer .......................................................................................................................................5
References ....................................................................................................................................................6
About Author ................................................................................................................................................7
Connect With US...........................................................................................................................................8

What does a class loader do?
In short and simple ….it locates and loads the class files.
A class loader is an object that is responsible for loading classes. The class ClassLoader is an abstract
class. Given the name of a class, a class loader should attempt to locate or generate data that
constitutes a definition for the class
Class not found issue
I wrote an article in Feb’10 about the class loader in websphere. Can be read here
Troubleshooting
You can use websphere application server’s in-built class loader viewer to examine the class loaders
visible to applications [war/ear].
To view the class loader view .. Navigate to Troubleshooting -> class loader viewer. Here you can see the
enterprise application topology which shows the application available under the servers.
To view a class loader, select a module within an enterprise application in the topology. For this tutorial
I’m using the default application [defaultWebApplication.war]. So click on the
defaultWebApplication.war . On the resulting page you can see 3 options.
Export - Opens a dialog that enables you to view or save the class loader information on this page in an
XML file.
Table view - Displays the Table view page, which provides information about class loaders visible to the
module in an HTML table format for each class loader.

Search – Display the search page, on which you can search class loaders for specific strings, jar files etc..

Class loader viewer
Note: you must enable class loader viewer to use it.
This displays the hierarchy of the class loader that affects the loading of classes used by the web/ejb
module.
The Hierarchy tab displays the class loaders in a tree hierarchy. Expand a hierarchy of class loaders to
view the following:
 Class loader names
 Arrows that point upwards beside class loader names, indicating that requests can go to a
parent class loader only and not go to a child class loader
 The names of classes that are loaded by a class loader
 The paths of property files and .jar files used by the classes
Click on Classes to view a list of classes loaded by a class loader
The Search Order tab lists the class loaders in the order in which the runtime environment uses them to
find and load classes.

References
1. About Class not found error :
http://websphereapplicationservernotes.wordpress.com/2012/03/21/classnotfound-a-close-
look-at-class-loader-in-websphere-application-server/
2. Basics of java class loaders : http://www.javaworld.com/javaworld/jw-10-1996/jw-10-
indepth.html
3. Understanding java class loaders : http://www.ibm.com/developerworks/java/tutorials/j-
classloader/

About Author
Joseph Amrith Raj is a technical consultant for multiple WebSphere products. He worked on various
product consulting and support teams including WebSphere Application Server, WebSphere MQ,
WebSphere Message Broker, Enterprise Service Bus and WebSphere Process Server. He has 7 years of
experience in administration, troubleshooting, consulting and he has significant experience in
architecture, strategy and leadership positions. He is IBM certified for WAS, WMQ, WPS , SOA and Cloud
Computing.

Connect With US
 Be Connected!

Joseph Amrith Raj
Tool -3: Thread and Monitor Dump Analyzer for Java

J o s e p h A m r i t h R a j ’ s W e b S p h e r e L i b r a r y Page 2
Contents
About TMDA ........................................................................................................................................................................3
Obtain and Install the tool................................................................................................................................................3
Opening a thread dump...................................................................................................................................................5
Analyzing the Dump...........................................................................................................................................................7
Thread Detail..................................................................................................................................................................8
Finding blocked thread details ...................................................................................................................................9
Thread aggregation................................................................................................................................................... 10
Comparing threads..................................................................................................................................................... 11
O!, that’s nice but when to use this tool? ..................................................................................................................... 12
Alright, this tool is all to do with Java thread dumps. How do I get those thread dumps?.............................. 13
References......................................................................................................................................................................... 14
About Author..................................................................................................................................................................... 15
Connect With US.............................................................................................................................................................. 16

In this third part of this series, we will be discussing how to diagnose issues with the java threads like: Hung
threads, Deadlocks, Resource starvation and server crash. IBM has developed a tool named “Thread and
Monitor Dump Analyzer for Java” also known as TMDA. TDMA, is a part of the ISA tool.
About TMDA
The IBM Thread and Monitor Dump Analyzer for Java (TMDA) analyzes java cores and diagnoses monitor
locks and thread activities to identify the root cause of hangs, deadlocks, and resource contention or
monitor bottlenecks. It compares each java core and provides process ID information for threads, garbage
collection frequency, allocation failure frequency, and a list of hang suspects.
Obtain and Install the tool
 Start the ISA
 Go to update  find new  tools add-ons
 Expand the JVM based tools from the results and select [tech preview] IBM Thread Monitor and
Dump analyzer for Java {TMDA} and install the tool.
 Restart the ISA
Click on Analyze problem and select TMDA and launch the tool.

Tools starting page

Opening a thread dump
 Go to file open ThreadDumps
 Select multiple thread dump files
 Selecting a file from the list, shows the details/summery about that thread dump.

You can find the following information from this page:
 Cause of the thread dump
 Operating system information
 Java information
 Class path
 User arguments
 Heap details
 Memory segment analysis
 Thread status analysis
 Thread method analysys
 Grabage collection details and history

Analyzing the Dump
Select the thread dump from the list and click on Analysis. You can do the following analysis
 Native memory analysis
 Thread detail analysis
 Details of the thread monitor
 Compare threads from different dumps

Thread Detail
Select a javacore file from the list and go to Analysisthread detail
Color codes are used to show the status of a thread status. Below is a table of the thread status codes.

Finding blocked thread details
Click on the thread that is shown as blocked from the thread list. On the right side panel, you can see the
blocked thread information.
Click on the entry in Blocked by list, shows the blocked thread

Thread aggregation details can be found by selecting a dump file from list and clicking on the thread
analysis.

Comparing threads
Select multiple thread dumps from the list and click Analysiscompare threads.

O!, that’s nice but when to use this tool?
 To Analyze Java Core files
 To diagnose monitor locks
 To find out java thread activities
 To find out and diagnose hangs in threads
 To diagnose deadlocks
 To diagnose resource bottlenecks/resource starvation

Alright, this tool is all to do with Java thread dumps. How do I get those thread
dumps?
If you get unexplained server hangs under WebSphere, you can obtain, from the WebSphere server, a
thread dump to help diagnose the problem. Thread dumps can either be generated automatically or
manually.
In the case of a server hang, you can force an application to create a thread dump.
On unix/Linux machines find the process id (PID) of the hung JVM and issue kill -3 PID. Look for an output
file in the installation root directory with a name like javacore.date.time.id.txt.
Or you can use wsadmin prompt,
 get the handle of the server
wsadmin>set jvm [$AdminControl completeObjectName type=JVM,process=server1,*]
 execute
wsadmin>$AdminControl invoke $jvm dumpThreads
If an application server spontaneously dies, look for a file. The JVM creates the file in the product
directory structure, with a name like javacore[number].txt.
From WebSpere Application Server v8 onwards.. You can generate these dumps using administration
console as well.

References
1. IBM Thread and Monitor Dump Analyzer for Java (Tech Preview)
2. How to Diagnose Java Resource Starvation

About Author
experience in administration, troubleshooting, consulting and he has significant experience in architecture,
strategy and leadership positions. He is IBM certified for WAS, WMQ, WPS , SOA and Cloud Computing.

Connect With US
 Be Connected!

Joseph Amrith Raj
Tool - 4: Garbage collection and
memory visualizor

Tool - 4: Garbage collection and memory visualizor
J o s e p h ’ s W e b S p h e r e L i b r a r y Page 2
Table of Contents
About GC and Memory Visualizer................................................................................................................3
Using GC and Memory Visualizer ................................................................................................................3
Obtaining and installing the tool.................................................................................................................4
Enabling Verbose GC ...................................................................................................................................4
Analyze Verbose GC log ...............................................................................................................................5
Customizations and Navigation ..................................................................................................................7
Axes Tab....................................................................................................................................................7
Zoom tab...................................................................................................................................................7
Keys panel................................................................................................................................................8
Templates.................................................................................................................................................9
Menu and Customizations.................................................................................................................... 10
Analyze and interpret the data................................................................................................................. 12
Report .................................................................................................................................................... 12
Table data.............................................................................................................................................. 12
Structured Data..................................................................................................................................... 13
Line plot................................................................................................................................................. 14
Cool … any real-time usage or examples?............................................................................................... 15
References ................................................................................................................................................ 16
About Author.............................................................................................................................................. 17
Connect With US ....................................................................................................................................... 18

In this part, we will be discussing how to obtain and diagnose Garbage collection and Memory
related issues in WebSphere application server using Garbage collection and Memory Visualizer.
About GC and Memory Visualizer
Garbage Collection and Memory Visualizer is part of the IBM support assistant workbench. This tool
can visualize the garbage collection data from IBM JRE 1.4.2 or higher. This tools helps you get
insights into garbage collection using graphs with data like Heap size, used heap size, pause times,
Using GC and Memory Visualizer
The GC and Memory Visualizer allow you to visualize your garbage collection data, as raw datasets,
line plots, reports, and images. The “Line plot” portion of the tool displays graphs of over forty
different garbage collection data characteristics – including used heap, pause times, and the reason
for garbage collection being triggered. You can read in multiple sets of garbage collection logs and
display them together on a single set of axes, which allows you to easily compare garbage collection
behavior across multiple test runs of an application. The “Report” section of the tool contains a
summary of the line plot data, along with information on general garbage collection behavior and
heap sizing recommendations based on overall heap occupancy.
The tool can provide the following:
 a graphical display of a wide range of verbose GC data values
 tuning recommendations and detection of problems such as memory leaks
 report, raw log, tabulated data and graph views
 saving of data to HTML reports, jpeg images or .csv files (for export to spreadsheets)
 viewing and comparing multiple logs
 analysis of optthruput, optavgpause, and gencon GC modes

Obtaining and installing the tool
 Start the IBM support assistant tool
 Go to update  find new  tools add-ons
 Expand he JMV based tools from the results and select “IBM Monitoring and Diagnostic Tools
for Java – Garbage collection and Memory Visualizer” and proceed to install.
 After installation is complete, restart ISA.
To use this tool, you need to have the GC logging information, which can be collected by enabling
verbose GC.
Enabling Verbose GC
In the WebSphere Application Server administration console:
 Servers  server_name  Server Infrastructure  Java Process Management process
definition  Additional properties Java Virtual Machine and enable “Verbose Garbage
Collection” check box
 Restart the server on which verbose GC is enabled.
Output will be written to the native_stderr. Log

Analyze Verbose GC log
Start the ISA and go to Analyze problems
Select the tool in the list and click launch
A pop-up window will ask you to provide the Log file which has the Garbage collection data.
Click on Browse  Remote Artifact Browse and select the log file

Click next and the tool will automatically analyze the log file and shows you data set from the input
log file. The default setting is to show the line plot.

Customizations and Navigation
You can select the X and Y axis parameters for the line plots (graphs). Check on the right side panel
of the line plot. You will find Axes tab and Zoom tab.
Axes Tab
Use this tab to customize the X and Y axes parameters.
X Axis is generally time which can be customized to msec, seconds, minutes, hours, days. And X axis
can be set to GC count as well.
Y axis is generally Size, which can be customized to bytes, KB, MB, GB and percentage.
Zoom tab
Use this to customize the zooming on either x-axis or y-axis or both.

The other way to zoom is to select area of interest on the graph, by holding the mouse left-click.
Keys panel
The keys panel shows what variants and data are being displayed in the graph. You can customize
the colors of the data on the graph. Click on the data being shown the data panel to change color of
that data on the graph.

Templates
This panel lists the templates available to plot the graphs. Selecting [double-click]a template from
the list shows the relevant graph.

Menu and Customizations
Different options are available on the menu to plot the customized graphs
Clicking in VGC heap/ VGC pause will display the available data options

Analyze and interpret the data
The following options can be used to understand the data from the analyzed VGC log.
Report: By default report shows tuning recommendations, summary heap size and used heap size
graphs. You can customize the report using the VGC menu item. The other options in the VGC menu
for report are GC reason, Classes loaded, JVM restarts etc...
Table data: Shows data from the GC log in table format. Defaults are GC count, used heap and
Heap size. You can customize this tabular representation of the data by selecting other data
parameters from either VGC pause or VGC heap menu items.

Structured Data: shows the summary from the analyzed GC log.

Line plot: shows the graphical representation of the data using the selected template or
customized options.

Cool … any real-time usage or examples?
 You can use this tool to diagnose Memory Leaks. If the used heap line is creeping up when
there's no obvious reason for the memory requirements of the application to be increasing,
there may be a leak. The GC and Memory Visualizer looks for this pattern and adds a
comment to the tuning recommendation if it detects something that is likely to be a leak.
However the tool will not show you which object are causing the leak.
 You can use this tool to determine what will be optimized Heap size for your applications. If
the heap is so small that the data required by the application will not fit into it, then the
application will run out of memory and terminate with an OutOfMemoryError. If the heap has
room for the application data but not much room to spare, the garbage collector will have to
spend a lot of time ensuring that there is room in the heap for new allocations, and this will
hurt application performance. A heap that is too big usually won't have a negative effect on
application performance, but it is wasteful, and GC pauses may be long.
 You can also use this tool to estimate the application throughput and response times. You
can change the GC policy to different available options and analyze which policy is giving best
response times. But it always recommended collecting the GC data when the application is
under good load.

References
1. Garbage Collection policies: part-1 and Part-2
2. Memory leak detection and analysis : part-1 and part-2

About Author
WebSphere Message Broker, Enterprise Service Bus and WebSphere Process Server. He has 7 years
of experience in administration, troubleshooting, consulting and he has significant experience in
architecture, strategy and leadership positions. He is IBM certified for WAS, WMQ, WPS , SOA and
Cloud Computing.

Connect With US
 Be Connected!

Joseph Amrith Raj
Tool -5: Visual Configuration Explorer

Contents
About VCE..........................................................................................................................................................................3
Obtaining and Installing VCE...........................................................................................................................................4
Using VCE............................................................................................................................................................................5
Step-1: Collecting data.................................................................................................................................................5
Step-2: Visualize the data.............................................................................................................................................8
Creating/Viewing Diagrams...........................................................................................................................................11
Understanding VCE user interface..............................................................................................................................13
Configuration View:....................................................................................................................................................13
Diagram view: ..............................................................................................................................................................14
Properties view:...........................................................................................................................................................14
Understanding the tool bar of providers view .........................................................................................................15
Search.............................................................................................................................................................................15
Compare .......................................................................................................................................................................16
It looks good, how can I use it?....................................................................................................................................17
How can I collect the data from remote machines? ...............................................................................................18
References.........................................................................................................................................................................19
About Author...................................................................................................................................................................20
Connect With US............................................................................................................................................................21

About VCE
VCE provides a way to visualize, explore and analyze the configuration information from different
sources. Visual Configuration Explorer lets you:
 Compare different snapshots of the same runtime environment.
 Create diagrams that show configuration information in a graphical format, including key
relationships between configuration elements.
 Access detailed configuration properties.
 Search across and within configurations for properties.
 Compare configurations.
 Correlate configuration information between different products.
 Save and restore your work.
 Collaborate with others during problem determination.
 Work in a standalone or client/server environment.

Obtaining and Installing VCE
VCE is an add-on in IBM Support Assistant (ISA). First make sure that you have ISA installed on your
system. IF ISA is not installed, refer to part-0 of this series which discuss how to get & install ISA.
 Start the ISA
 Go to upatemanage installed add-ons installed product add-ons
 Make sure, the product for which you want to collect and visualize the data is active/enabled.
When it is enabled, you don’t see the red icon. If the product you choose is not enabled, right-
click on the product name and enable it and restart ISA.
 After enabling the products, for which you like to use the VCE, go to update find new tools
add-ons others
 Select and install Visual Configuration Explorer and restart ISA. With this you’ve successfully
completed the installation of ISA.

Using VCE
VCE works in two steps
 Step-1: collect the data
 The collector will collect the data and generate a VCE compatible output file
 Step-2: visualize the data
 You import the VCE output file to the workspace and get the visual diagram.
Step-1: Collecting data
Start the ISA toll and click on Analyze problem Collect Data
In the collectors section, select the product for which you want to collect the data. I’ve WAS v8 on my
machine, so I’ll be collecting data for it. Expand the product selection  general
 Select VCE collector and click on add

 Now you can see it in the Collector queue
 Then click on Collect All
 It will ask you for WebSphere Application Server root directory. Browse and select

 After you click OK, it’ll start collecting data.
 You can see the progress in the collector status
 If you have multiple profiles in your websphere application server installation, it will ask you to
choose the profile.

 Select a profile and click OK
 If you have multiple servers, it will ask you to choose for which server you like to collect the
data.
 If the security is enabled, it will you for credentials
 Once the data collection is completed, it asks you if you want to run VCE. Select YES
 It may ask you, if you to like to upload the data to IBM. Select No
Wait until the status Collector status is changes to complete. Once completed, it will show you the
collected zip file in the collected result section. Click on View details and note down the file location.
Step-2: Visualize the data
 Click on the tools tab or go back to home analyze problem
 Select Visual configuration explorer and laucn the tool

 It will show you a welcome page

 Go to File  visual configuration explorer  new workspace, to create a new workspace
 After that click on the + icon to add the collection
 Now give a name for the collection and browse for the file which was created in step-1.
 Click OK. Now you can see it listed in the configurations section.
So now the configuration is successfully loaded.

Creating/Viewing Diagrams
Right-click on a configuration object and select ‘create diagram’
For illustration purpose, I am creating application servers diagram. You can create diagrams for core
group, database resources, SI bus, Messaging resources etc..
Create diagram  Show application Servers
You can create the following diagrams for ‘applications’

Here is a sample deployment topology
You can export the diagrams as JPEG using Diagram  export diagram
image to JPEG option from the menu.

Understanding VCE user interface
Configuration View: this is known as providers view. Each provider represents one configuration
source and information gathered from it. This view lets you see all the configurations currently loaded.
Each provider is displayed as tree with hierarchy of objects, grouped based on the type. Each object in
the hierarchy has properties represented as a tree structure of name/value pairs. Topology relationships
describe the correlation between the objects.

Diagram view: this view lets you graphically view the relationship between the objects. You can
create the diagrams by right clicking an object in the providers view.
Properties view: The properties view shows the individual properties. The properties view can
also be used to compare multiple objects. This search functionality will be discussed later in this article.

Understanding the tool bar of providers view
Search
Select the entire configuration in the providers view and then click on the search button or right click
and select search. A window pops up showing search settings.
Enter a search string and click OK. For example, i like to know what all servers have HealthCenter
agent enabled in my cell, so I entered the string which enables the agent.

The results section shows results from the search. Single-click on a entry in search results automatically
display the object the providers view and double-click on a entry in search result shows the properties
of that object.
Compare
Using this you can compare the configurations of different topology objects. For example, if you collect
the configuration data every week from a websphere cell, you can compare using this option to see
what configuration changes have been done to the cell over the past week.
To use compare feature, select multiple configurations or objects in the providers view and click on the
compare button. A dialog box appears with three options: Subtree, Properties and Cancel. Selecting the
subtree option recursively compare the two complete configurations.
When compare is finished the results are shown in separate view.

The properties different between the two configurations are shown in RED color. Clicking on the RED
compare result shows both objects in the providers view and double-click will show the properties of
both the objects side-by-side.
It looks good, how can I use it?
Using VCE:
 You can visualize and explore configurations of many WebSphere/Lotus/Tivoli products.
 You can use search function to find out a property across a cell. For example, Maximum heap
size setting for each and every JVM in a cell.
 You can compare configurations of multiple servers. For example, if one of the members of the
cluster has issues, then you can compare the configuration with another member of that cluster.
 Diagram representation will speed up the process of understanding the configurations.
 You can visually see all database/messaging dependencies and how they are interacting.
 You can create application diagrams shows the deployment topology and dependencies
Above are some of the things you can do with VCE. Install it and explore, you’ll many more reasons to
have it.

How can I collect the data from remote machines?
You can collect the data from remote machines using ISA agents. This involves installing IBM support
assistant agent manager on the machine which has ISA and then installing IBM support assistant agents
on the target remote machines.
Full details and detailed instructions can be found here on IBM Education assistant

References
1. Configuring Remote Agents for data collection : IBM Edu Assistant
2. Get to know VCE : IBM Support Authority
3. Documentation: use the help button on the VCE tool.

About Author
architecture, strategy and leadership positions. He is IBM certified for WAS, WMQ, WPS , SOA and
Cloud Computing.

Connect With US
 Be Connected!

HTTP://FACEBOOK.COM/WEBSPHERELIBRARY
Joseph Amrith Raj
Integrated Tivoli Performance Viewer [TPV] in
WebSphere Appplication Server
This document shows how to configure and use WebSphere Performance Monitoring
Infrastructure [PMI] and Integrated Tivoli Performance Viewer [TPV]

1
IntegratedTivoliPerformanceViewer[TPV]inWebSphereAppplicationServer
In general, our websphere application serving environments consists of different web servers, websphere
application server, database server and several network components. Monitoring these systems is a very
important job to make sure that your application is available at all time and monitoring also helps to tune
the performance. In WebSphere application server, the core of the monitoring infrastructure is
‘Performance Monitoring Infrastructure’ known as PMI. The performance data provided by WebSphere PMI
helps to monitor and tune the application server performance. PMI provides a comprehensive set of data that explains
the runtime and application resource behavior. For example, PMI provides database connection pool size, servlet
response time, EJB method response time, JVM garbage collection time, CPU usage, and so on. This data can be used
to understand the runtime resource utilization patterns of the thread pool, connection pool, and so on, and the
performance characteristics of the application components like servlets, JSP, and enterprise beans
PMI must be enabled, before collecting any performance data. If PMI is enabled after the server is started, the server
needs to be restarted to start the PMI. By default PMI is enabled. After the PMI is enabled, monitoring of individual
components can be enabled or disabled dynamically. There are some predefined statistic sets that can be used to
enable a set of statistics.
Statistic set Description
None All statistics are disabled.
Basic Statistics specified in J2EE 1.4, as well as top statistics like CPU usage and live HTTP sessions
are enabled. This set is enabled out of the box and provides basic performance data about
runtime and application components.
Extended Basic set plus key statistics from various WebSphere Application Server components like
WLM, and dynamic caching are enabled. This set provides detailed performance data about
various runtime and application components.
All All statistics are enabled.
Custom Enable or disable statistics selectively.

2
IntegratedTivoliPerformanceViewer[TPV]inWebSphereAppplicationServer|
Enabling PMI
 To enable PMI, in the administration console
 Go to Monitoring and Tuning  Performance Monitoring Infrastructure (PMI)  server_name general
properties under configuration tab
 Select Enable Performance Monitoring Infrastructure (PMI).
 Click OK and Apply changes
 Restart the application server
PMI data can be monitored and analyzed by Tivoli Performance Viewer (TPV), other Tivoli tools, your own
applications, or third party tools. TPV is a graphical viewer for PMI data that ships with WebSphere Application
Server.

3
Integrated Tivoli Performance viewer
Using the TPV, you can view the current activity of System resources, WebSphere pools and customer application
performance data. You can also log the data.
Go to Monitoring and Tuning performance viewer current activity
Select a server and click on start monitoring.
Now click on that server name
A TPV console is displayed. On the left side navigation, you’ll have the following options
Option Description
Advisor Provides tuning advice based on the collected PMI data from your server
Settings Provides tuning advice based on the collected PMI data from your server

4
Summary Reports Predefined reports that allow you to quickly analyze your server’s
performance. The different reports that are available are described below:
Performance Modules List of available PMI modules on your server.
By default, the right hand side shows the servlets summery reports.
Summary reports
Servlets Displays total requests and average response time data for all of the Servlets that are
currently gathering PMI data.
EJBs Displays number of method calls and average response time data for all the Enterprise
Beans that are currently gathering PMI data. Data is grouped according to bean name.
EJB methods Displays number of method calls and average response time data for all the Enterprise
Beans that are currently gathering PMI data. Data is grouped according to method
name.
Connection
Pool
Displays pool size and percentage of pool in use for data sources that are actively
collecting PMI data.
Thread Pool Displays pool size and number of active threads for the different thread pools within
WebSphere.

5

6
Settings
User Settings
 Refresh Rate: How frequently TPV collects performance data from a server.
 Buffer size: Specifies the number of entries to be stored for a server. Data displayed in TPV is stored in a
short in-memory buffer. After the buffer is full, each time a new entry is retrieved the oldest entry is
discarded.
 View data as: Specifies how the counter values are displayed.
o Raw: display the absolute value.
o Change in value: display change in the current value from the previous
o Rate of change: displays the ratio of the change from present value to previous value.
Log settings

7
Performance Modules
Each module has several counters associated with it. These counters are displayed in a table underneath the data chart
or table. Selected counters are displayed in the chart or table. You can add or remove counters from the chart or
table by selecting or deselecting individual counters. By default, the first three counters for each module are shown.
You can select up to 20 counters and display them in the TPV in the Current Activity mode.
Under performance modules, select the modules you like to monitor and then click on the view modules button. For
the below example, I selected JDBC connections, JVM runtime, Thread pools and Web Applications. You can
expand each performance module to get the fine-grained details about a particular JDBC connection, web application
etc...
For fine-grained details about a specific module, expand the module by clicking the ‘+’ sing and select the module
you like to monitor.
 If you like to see the data in table rather than a graph, click on ‘View Table’ button. To go back to chart,
click on ‘View Chart’

8
 If you click ‘Clear Button’, the PMI data will be removed from a table or chart
 You can manually adjust the scale for each counter so that the graph displays meaningful comparisons of
different counters. Find the counter whose scale you want to modify in the table beneath the chart. Change
the value for Scale as needed and click update.
o When the scale is set to 1 the true value of the counter is displayed in the graph.
o A value greater than 1 indicates that the value is amplified by the factor shown.
o A value less than 1 indicates that the variable is decreased by the factor shown.
 Reset to Zero sets a new baseline using the current counter readings at the instant the button is clicked.
Future datapoints are plotted on the graph relative to their position at the time Reset to Zero is clicked.

9
Modify run-time PMI settings
Go to monitoring and tuning performance monitoring infrastructure (PMI) server_name
Click on the runtime tab
Here, you can change the currently monitored statistics set. Your options are None, Basic, Extended, All and
Custom. Click on the ‘+” button to see what details are monitored in a specific monitored statistics set.
If you select the ‘custom’ monitored statistics set, you can get the fine grained PMI data.
Select Custom and click on the custom.

10
Now click on module you like to monitor and on the right side panel, select the counter and click enable. For
example, you like to monitor Garbage Collection times. Click on JVM runtime  Garbage collection and select
‘GCtime’ and click enable.
Now go back to Monitoring and Tuning performance viewer current activity. Select the server.

11
You can see the module you selected now in the table/graph and being collected. Remember this change is
temporary and next time you restart the server, it will not be collected.

12
Recording and Replaying logs
 Monitoring and Tuning performance viewer current activity. Select the server.
 Selecting the performance modules whose data you like to log.
 Then click on one summery report [like servlet/ejb ]
 Then select ‘start logging’. A message will be displayed saying logging has been started for serverX on
nodeX
 After sometime, click on ‘stop logging’. A message will be displayed saying logging has been stopped for
serverX on nodeX
 Now go to Monitoring and Tuning  Performance viewer view logs
 Specify the log file you want to replay. By default, the log files are stored under
/WAS_HOME/profiles/profile_name/logs/tpv
 Click on ‘view log’ to replay the log.
 The log will begin replaying in the panel on the right.
Note: Navigate to the different Summary Reports. Notice that performance data was also recorded for these views as
well.
You can also select different performance modules and then click ‘view modules’ and start logging the data.

13
About Author
Joseph Amrith Raj is a technical consultant for multiple WebSphere products. He worked on various product
consulting and support teams including WebSphere Application Server, WebSphere MQ, WebSphere Message
Broker, Enterprise Service Bus and WebSphere Process Server. He has 7 years of experience in administration,
troubleshooting, consulting and he has significant experience in architecture, strategy and leadership positions. He is
IBM certified for WAS, WMQ, WPS , SOA and Cloud Computing.

14
Connect With US
 Be Connected!

Joseph Amrith Raj
Recover lost KDB password

Script to recover lost KDB password 2
This script is not my work; I got it on some forum back in 2008. So Credit and Thanks
goes to the actual author.
Scenario
Sometimes you may not be able to retrieve the password of the KDB file.
When you create a new KDB file using iKeyman utility, it asks you to create
a stash file. This stash file will come to your rescue when you lose KDB
password. You can use the following script against Stash file to recover
the password.
Script
#!/usr/bin/perl -w
#unstash.pl - "decrypt" IBM HTTP server stash files. No, really. They
<strong>are</strong> this pathetic.
use strict;
die "Usage: $0 <stash file>n" if $#ARGV != 0;
my $file=$ARGV[0];
open(F,$file) || die "Can't open $file: $!";
my $stash;
read F,$stash,1024;
my @unstash=map { $_^0xf5 } unpack("C*",$stash);
foreach my $c (@unstash) {
last if $c eq 0;
printf "%c",$c;
}
printf "n";

How to Use
1. Copy the above script and save it as unstash.pl
2. Change the permissions on the script to have execute permissions
3. ./unstash.pl STASH_FILE_NAME
Tip: if you get ‘n’ at the end of the output… do not copy ‘n’.

About Author
architecture, strategy and leadership positions. He is IBM certified for WAS, WMQ, WPS , SOA and Cloud
Computing.

Connect With US
 Be Connected!

IBM Support
Assistant v5
Review and hands-on by Joseph

What's new in v5
 This new version is built on top of WebSphere application
server community edition.
 It gives more flexible configurations
 IntuitiveGUI and access through browser
 Case Manager
 File Management
 Can be installed on server or on desktop
 Problem analysis will generate reports

New in v5
 Since it is based on WebSphere application server community
edition :
 You can manage the required memory details
 User management

New in v5
 ISA v5 can be installed on a server and multiple users can
access the same.
 ISA v5 can also be installed on your local desktop for single
user usage.

Using ISA Console
 Using the browser based ISA console, you can do
 Case management
 File management
 Data collection
 Problem analysis using tools

Case management
 Case management in
simple words is organizing
different problem analysis
files and reports.
 For example you are
troubleshooting two
different issues, then you
can create two different
cases case0001 and
case0002 and organize the
input files and output
reports into the cases.

File
management
•You can see all the files using ‘files’ tab
•This will let you easily and quickly find the
files
•You can see this option like ‘explore’ option
of the windows based operating systems.
•You can also perform several actions
against the files in the list.

Tools
 The tools tab lists the available tools for problem
analysis.
 The present set of tools available in ISA v5 are:
 Garbage Collection and Memory Visualizer
(GCMV)
 IBM Portal Log Analyzer
 Memory Analyzer
 Pattern Modeling and Analysis Tool for
Java Garbage Collector (PMAT)
 Profile Port Checker
 Thread and Monitor Dump
Analyzer WebSphere Application Server
 Configuration Visualizer

Data Collector
•Data Collector runs in your browser or on
your operating system and collects data
about your system or about a product of
your choosing.
•It provides the ability to run a collection
from your system in an interactive,
conversational mode and ultimately upload
the data to IBM.

How to get it
 ISA v5 tech preview can be downloaded from : http://www-
01.ibm.com/support/docview.wss?uid=swg27023689

Starting and stopping
 After downloading theTech Preview zip file:
 extract the file,
 Starting : run the start_isa.bat or start_isa.sh file
 go http://<hostname>:8080/isa5 , in browser
 Stopping : run the stop_isa.bat or stop_isa.sh file

WebSphere Application Server CE
 WAS CE console can be
accessed using
http://localhost:8080/cons
ole/portal/Welcome , after
starting the tool as
mentioned in previous
slide.
 Default username and
password :
system/manager.

Configurations for ISA v5
 ISA v5 uses port 8080. If you are already using that port, it can
be changed from “isa_rootvarconfigconfig-
substitutions.properties”
 This location of cases and files can be adjusted using a
configuration file located under “isa_rootisa.properties”
 “Isa_root” is the directory where you unzipped the tool.
 Make sure there are no spaces in the isa_root path.

Working with cases
 To access/manage the cases , click on the cases button.
 In the sliding popup you can create/delete/manage cases.
 Before you perform any other task, you must have one case
defined.
 First you select a case and then add files or do problem
analysis .
 Click cases  add , to create a new case.
 Use drop-down button to select the case.

Working with files
 After you select a case, you
can add files to it by either
using ‘addfiles’ or drag-n-
drop option.
 Different actions can be
performed directly from
files tab by right-click on
the file.
 You can use
troubleshooting tools
directly from the right-click
context menu.

Problem Analysys
 I have the verbose GC log and would like to analyze it for any patterns of
memory leaks.The steps are :
 Create a new case for this investigation, using cases button.
 Import the verboseGC log into the case using ‘files’ tab.
 Right click on the imported file  problem analysis  pattern modeling
and analysis tool for java garbage collector [PMAT]  submit
 A pop-up shows the analysis output directory. Click OK
 Wait for the task to finish
 You can see the output directory in the files panel. If you click on that, you
can see the files in it.
 Open the HTML to see the analysis results.

Cases and files
When you are using default
configurations, the cases directory will
be automatically created under
“isa_rootisacases”
This location can be adjusted using a
configuration file located under
“isa_rootisa.properties”

References
 ISA v5 offline documentation/readme [found under ISA_root/doc]
 ISA v5 documentation
 Problem determination tools for ISA v5

Contact
About
 Joseph, is a technical consultant for
multiple WebSphere products. He has
7 years of experience in
administration, troubleshooting and
consulting. He has significant
experience in architecture, strategy
and leadership positions
Contact
 Twitter
 Facebook

ThankYou

WEBSPHERE DOCTOR – WEB SERVER TROUBLESHOOTING
Joseph’s WebSphere Library 1
The following has been written based on the assumption that you are using either IBM http
server or Apache in front of your WebSphere application server.
So… how to troubleshoot issues with web server?
 First and most important thing is to know, if the web server is really running. If you are
on unix machines check for httpd process using ps command and also look for pid file
under the logs directory. On windows machine you can check in the task manager for the
process.
 Next thing is log files. There are two log files in the web servers: access log and error log.
Access log contains the logging of who accessed what on that web server. Error log
contains the start-up messages and any error messages. It is important to note that, when
you are using virtual-hosts in the web servers, each virtual host will have its own access
and error logs.
 Next important thing you need to remember are the HTTP response codes. These codes
are explained here in detail. Until you get familiar with these response codes.. you can
take a printout of this image and place it in your desk.
Tips
 I generally receive emails from readers asking.. How do you troubleshoot a 404 error or
500 error. A simple answer for 404 is, since 404 is page not found, check if the request
url is correct.. if so then check if the content is available. Similarly for 50x, it could be a
configuration issue or your back-end server didn’t responded. In these kind of issues the
above image will help you get an idea where to look for.
 The other good idea is to enable the “server-status” option in the web server. But when
you enable this option you need to keep in mind that, you need to restrict the access to
this page. More details about server-status and how it can help can be found here.
 Another useful tool is , apachectl command. Using this you can test a new configuration,
check the syntax, temporarily change log level etc.. Some of these are discussed in our
Admintoolkit post, here.
While troubleshooting any issue… first make sure that the first component that is getting the
requests is working fine. So before jumping onto WebSphere plugin and application servers…
make sure web server is working as expected.

WEBSPHERE DOCTOR – WEB SERVER TROUBLESHOOTING
Http response codes

WEBSPHERE DOCTOR – PLUG-IN
Contents
Plug-in log file............................................................................................................................................2
Plug-in log format .....................................................................................................................................2
Web Server access log ..............................................................................................................................3
Other Tools ...............................................................................................................................................4
LSOF.......................................................................................................................................................4
Netstat ..................................................................................................................................................4
Telnet ....................................................................................................................................................4

Plug-in log file
This log file can be used to troubleshoot: plug-in initialization failures, HTTP errors, application server
hangs, unavailable application servers, network problems, timeout errors in a Web browser
If you think that the error is at plugin level, the log file can be found at the following location
 Default: WAS_ROOT/logs/http_plugin.log
 Or you can find the path from plugin-cfg.xml file
In the plugin configuration file, you can find the log level also. LogLevel can be Error/WARN/TRACE
<Config>
<Log LogLevel=”Error" Name="/apps/WebSphere/logs/http_plugin.log"/>
The default log level is “Error”
When you change the plugin log level, web server restart is not required since the plugin configuration
file will be reloaded based on the RefreshInterval variable value.
Plug-in log format
[ddd mm dd hh:mm:ss yyyy] PID TID - MsgType: Component: Task: Message
 PID is the process ID of web server in hexadecimal format
 TID is the thread ID of a specific request within that PID in hexadecimal format
 MsgType can either be PLUGIN or it can be one of the three LogLevels.
o When MsgType is PLUGIN, it means the accompanying message is an informational
message about the normal operation of the plug-in.
o When MsgType is one of the three log levels, the message is an ERROR, WARNing or
a TRACE statement.

Web Server access log
This file logs every response going back to the Web browser. If a Web browser initiates a request and
never gets a response back because it is indefinitely hung inside the plug-in or the application server,
you will never see a corresponding response entry of that request in this file.
 If you see time gaps in access.log where you should be seeing a steady stream, it may imply that
your users are not getting successful responses, at least not at the rate at which you expect.
 If you see response codes other than 200 building up in access.log, it means that your system is
deviating away from a healthy state.

Other Tools
LSOF: when used with –I option, this command will show all the open files and connections. This will be
a good starting point to know where the connections are stuck.
Netstat : this command will help when you suspect a network issue. This command output can show
you all the connection on a particular port/IP and their status
Telnet: this command can be used as a troubleshooting tool to know, if a port is accepting any
connection or if there is any delay.

Websphere doctor - your guide to diagnose issues

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Websphere doctor - your guide to diagnose issues

Similar to Websphere doctor - your guide to diagnose issues (20)

Recently uploaded

Recently uploaded (20)

Websphere doctor - your guide to diagnose issues