Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

WebSphere Technical University: Top WebSphere Problem Determination Features

Problem determination is an important focus area in the IBM WebSphere Application Server. Serviceability improvements have been added that have greatly improved the ability to find root causes of problems in both the full IBM WebSphere Application Server profile, and the newer Liberty profile. The session focuses on how to effectively use serviceability improvements added to the application server since V8.0. This includes high performance extensibe logging, cross-component trace, IBM Support Assistant data collector, timed operations, memory leak detection/prevention, and IBM Support Assistant 5.

Presented at the WebSphere Technical University 2014, Dusseldorf

  • Be the first to comment

WebSphere Technical University: Top WebSphere Problem Determination Features

  1. 1. A6: Top IBM WebSphere Application Server Problem Determination Features Chris Bailey STSM, IBM Runtime Monitoring and Diagnostics
  2. 2. Please Note IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here. © 2014 IBM Corporation 2
  3. 3. Introduction to the Speaker Chris Bailey STSM, IBM Runtime Monitoring and Diagnostics Architect 14 years experience developing and deploying Java SDKs 1 years experience developing and deploying Node.js SDKs 3 Recent work focus: Java and Node.js monitoring and diagnostics Java and Node.js integration into the cloud Highly resilient and scalable deployments Requirements gathering Contact Details: baileyc@uk.ibm.com http://www.linkedin.com/in/chrisbaileyibm http://www.slideshare.net/cnbailey/ @Chris__Bailey
  4. 4. 4 Problem Reduction 1. Eliminate Problems remove the source of problems so that they no longer occur. For example, redesign a component to eliminate the need to configure something that customers have trouble with. 2. Improve Serviceability make it easier for customers or support to service problems when they occur. For example, provide a way for customers to quickly check to see if there are any configuration problems. Eliminate Problems Improve Serviceability Goal: Reduce total problem hours
  5. 5. 5 Problem Reduction Serviceability Features • Hang detection • First Failure Data Capture • High Performance Extensible Logging • Cross Component Trace • Memory Leak Detection and Prevention • Timed Operations • … Component Improvements • Security • Systems Management • Install • Plugin • Java™ and JDK • WebServices • ... Problem Determination Tools • IBM Support Assistant Team Server (with Automated Analysis) • Garbage Collection and Memory Visualizer • Memory Analyzer (with WAS extensions) • Java Health Center • Thread and Monitor Dump Analyzer • Performance Tuning Toolkit • ... Problem reduction is driven from multiple directions  Component improvements reduce problem determination time for key components  Serviceability features simplify problem determination across the product  Problem determination tools make it easier to analyze complex issues
  6. 6. WAS V8.5.5 (L) Binary Logging (L) Timed Operations (L) Sensitive WAS Runtime Features Log and Trace Guard WAS V8.5.x (F) Cross Component Trace ISA DC integration (F) HPEL enhanced filtering (F) Memory leak detection and protection (L) Liberty Serviceability (logs, dumps, FFDC) 6 WAS V5.1.x Session data crossover assist Socket Connect Timeouts High Performance Extensible Logging (HPEL) JVM Serviceability Enhancements WAS V6.1.x (*) Diagnostic Providers Unique Message IDs Trigger dumps from console Sensitive log and trace guard WAS v6.0.x JSR 47 Logging Common Base Events FFDC Aspects FFDC Capability for z/OS Class Loader Viewer WAS V5.0.x FFDC Collector Summary Config Validation Connection Manager Serviceability Hang Detection in the Runtime WAS V8.0.x WAS V7.0.x (*) FFDC Improvements Diagnostic Tooling Framework for Java (DTFJ) Ongoing Serviceability Defects Process to report and address specific serviceability issues encountered in the field (*) Several tooling-related projects were delivered as WAS Line Items in WAS 6.1 and 7.0 (F)=Full Profile (L)=Liberty Profile
  7. 7. 7 Liberty
  8. 8. Liberty Problem Determination Server dump Command Server javadump Command TimedOperations Feature Binary Logging © 2014 8 IBM Corporation
  9. 9. Server Dump Command  The server dump command is a utility you can run from the bin directory to dump and zip up important state information from the application server itself. wlpbin> server dump  The server dump command collects: – Log files – Configuration information – Details of deployed applications  If the server is running when the command executes, it also collects: – State of each OSGi bundle in the server – Wiring information for each OSGi bundle in the server – Component list managed by the Service Component Runtime (SCR) environment – Detailed information of each component from SCR – Configuration administration data of each OSGi bundle – Information about registered OSGi services – Runtime environment settings such as Java™ virtual machine (JVM), heap size, operating system, thread information, and network status © 2014 9 IBM Corporation
  10. 10. Server JavaDump Command  The server javadump command is a utility you can run from the bin directory to collect javacores, heap dumps, and system cores. wlpbin> server javadump – Collects a javacore from a running server. Javacores are helpful for diagnosing hangs, 100% CPU conditions, and deadlocks. Use tools such as the Thread and Monitor Dump Analyzer available in the IBM Support Assistant to analyze javacores. wlpbin> server javadump –-include=heap – Collects a heap dump from a running server. Heap dumps are helpful for diagnosing memory leaks. Use tools such as Memory Analyzer available in the IBM Support Assistant to analyze heap dumps. wlpbin> server javadump --include=system – Collects a system core from a running server. System cores are helpful for diagnosing memory leaks, or exploring specific data in your heap address space. Use tools such as Memory Analyzer, or Interactive Diagnostic Data Explorer available in the IBM Support Assistant to analyze system cores. © 2014 10 IBM Corporation
  11. 11. Timed Operations  Timed operations is a new feature that tracks the duration of JDBC operations running in the application server –Logs a warning when operations take more or less time to execute than expected –Periodically creates a report in the application server log, detailing which operations took longest to execute – When running the server dump command, it will generate a report containing information about all operations it has tracked –The information listed in these reports can be used to decide if anything has an unusual behavior –Recommended for production environments to track slowdowns when they first occur © 2014 11 IBM Corporation
  12. 12. Timed Operations  Sample logged message [3/14/13 14:01:25:960 CDT] 00000025 TimedOperatio W TRAS0080W: Operation websphere.datasource.execute:jdbc/social:INSERT INTO DB2INST1.CommentTab (comment, extension, object, time, userid, mentionsid) VALUES (?, ?, ?, ?, ?, ?) took 16.613 ms to complete, which was longer than the expected duration of 10.859 ms based on past observations.  Sample automatically generated report in the log © 2014 12 IBM Corporation
  13. 13. Timed Operations  Sample introspection file The description of this introspectable service: Information about the timed operations, grouped by type, and sorted within each group by expected duration --------------------------------------------------------------- Timed operations for: websphere.datasource.execute --------------------------------------------------------------- Expected duration (ms) Standard deviation (ms) Timed operation 1.204297 0.929251 websphere.datasource.execute:jdbc/TradeDS:update orderejb set holdingID = ? where orderid = ? 1.107012 1.127067 websphere.datasource.execute:jdbc/TradeDS:delete from orderejb 0.286977 0.163669 websphere.datasource.execute:jdbc/TradeDS:update orderejb set holdingid=null where holdingid = ? 0.211317 0.083466 websphere.datasource.execute:jdbc/TradeDS:delete from accountejb 0.209597 0.243975 websphere.datasource.execute:jdbc/TradeDS:update orderejb set orderstatus = ? where orderid = ? 0.201933 0.059850 websphere.datasource.execute:jdbc/TradeDS:select * from orderejb o where o.orderstatus = 'closed' 0.144153 0.052846 websphere.datasource.execute:jdbc/TradeDS:delete from holdingejb where holdingID = ? 0.143007 0.037623 websphere.datasource.execute:jdbc/TradeDS:update orderejb set holdingID = ? where orderid = ? 0.139108 0.042636 websphere.datasource.execute:jdbc/TradeDS:update accountejb set lastLogin=?, where userID=? 0.137844 0.125086 websphere.datasource.execute:jdbc/TradeDS:delete from orderejb where orderStatus='cancelled' 0.128342 0.042986 websphere.datasource.execute:jdbc/TradeDS:delete from accountprofileejb 0.114781 0.029487 websphere.datasource.execute:jdbc/TradeDS:update accountejb set balance = ? where account = ? 0.028868 0.128856 websphere.datasource.execute:jdbc/TradeDS:select * from orderejb o where o.orderid = ? 0.021680 0.008652 websphere.datasource.execute:jdbc/TradeDS:select * from quoteejb q where q.symbol=? © 2014 13 IBM Corporation
  14. 14. Enabling Timed Operations  To enable timed operations add the following feature to your server.xml file: <feature>timedOperations-1.0</feature>  To disable timed operations remove this line from your server.xml file  A server restart is not required to enable or disable timed operations © 2014 14 IBM Corporation
  15. 15. Configuring Timed Operations  To disable the generation of the report to the logs <timedOperation enableReport="false"/>  To change the frequency of the report, for example to once every 12 hours <timedOperation reportFrequency="12"/>  To log a warning when the total number of timed operations reaches the specified value <timedOperation maxNumberTimedOperations="10000"/>  To enable automatic cleanup of least recently used timed operation data from memory (freeing up space as needed for new operations to be tracked), add the following to your server.env file: com.ibm.timedOperations.autoCleanup=true © 2014 15 IBM Corporation
  16. 16. Liberty and Full Profile 16
  17. 17. Problem Determination Capabilities in Both Liberty and Full Profile Binary Logging (Liberty) High Performance Extensible Logging (Full Profile) © 2014 17 IBM Corporation
  18. 18. Binary logging / High performance extensible logging High speed log and trace handling for WebSphere Application Server –Available in full profile since WAS 8.0 –Referred to as High Performance Extensible Logging (HPEL) –Available in Liberty since WAS 8.5.5 - Referred to as binary logging © 2014 18 IBM Corporation
  19. 19. Binary logging / High performance extensible logging Benefits –Greatly improves speed of logging and tracing –Enables you to add new fields to log and trace records –Helps you filter through logs and trace with a powerful command line tool –Provides a common solution for all supported platforms –Includes an API to read from binary repositories programmatically  Notes – Stores log and trace data in a binary format – Does not require any change to your application code – Able to work with log analytics tools © 2014 19 IBM Corporation
  20. 20. Runtime Logs – Full Profile Full Profile - default logs trace System.out System.err SystemOut.log SystemErr.log trace.log activity.log1 Full Profile - HPEL logs trace System.out System.err logdata/ tracedata/ TextLog2 3 1activity.log is deprecated and can be disabled. 2TextLog is redundant and can be disabled if you don't need a plain text log 3Inclusion of trace content in TextLog is optional and not recommended for high performance © 2014 20 IBM Corporation
  21. 21. Runtime Logs – Liberty Liberty - default logs trace System.out System.err messages.log trace.log console.log1 Liberty – binary logging logs trace System.out System.err logdata/ tracedata/ console.log1 1console.log is redundant and can be disabled if you don't need it © 2014 21 IBM Corporation
  22. 22. Viewing logs and trace  Viewing logs and trace – [Liberty] binaryLog command is used to view content from logdata / tracedata – [Full Profile] logViewer command is used to view content from logdata / tracedata – displays repository content in plain text – provides options to filter the displayed content and to choose the output format – can be used to monitor (tail) the repositories in real time © 2014 22 IBM Corporation
  23. 23. Binary Logging Performance (Liberty) – Measured using java.util.logging – Much faster than logging to messages.log when console.log was disabled – Much faster than tracing to trace.log – Performance gains are very dependent on log / trace content  Use binary logging to reduce performance impact of logs and traces on your production servers – console.log should be disabled when superior logging performance is required (note that this is not a factor in trace performance) <logging consoleLogLevel="OFF" /> Liberty Trace Performance 0 2 4 6 8 10 12 14 16 18 700000 600000 500000 400000 300000 200000 100000 0 Default HPEL threads events per second  Performance Liberty Logging Performance 0 2 4 6 8 10 12 14 16 18 700000 600000 500000 400000 300000 200000 100000 0 Default (with console.log) Default (no console.log) HPEL (with console.log) HPEL (no console.log) threads events per second © 2014 23 IBM Corporation
  24. 24. HPEL Performance (Full Profile)  Performance – Measured using java.util.logging – Much faster than logging to SystemOut.log when TextLog and JMX Notification are disabled – Much faster than tracing to trace.log – Performance gains are very dependent on log / trace content  Use HPEL to reduce performance impact of logs and traces on your production servers Full Profile Logging Performance threads events per second 0 2 4 6 8 10 12 14 16 18 800000 700000 600000 500000 400000 300000 200000 100000 0 Default Default (no JMX Notification) HPEL HPEL (no JMX Notification, no TextLog) Full Profile Trace Performance 0 2 4 6 8 10 12 14 16 18 700000 600000 500000 400000 300000 200000 100000 0 Default HPEL threads events per second © 2014 24 IBM Corporation
  25. 25. Enabling binary logging (Liberty)  To enable binary logging add the following line to your bootstrap.properties file in your server root directory: websphere.log.provider=binaryLogging-1.0  To disable binary logging remove the above line from your bootstrap.properties file  A server restart is required to enable or disable binary logging © 2014 25 IBM Corporation
  26. 26. Enabling HPEL (Full Profile) Administrative Console 1. Expand Troubleshooting 2. Select Logs and trace 3. Select server1 4. Select Change log and trace mode 5. Click Switch to HPEL Mode 6. Save your configuration 7. Restart your server  A server restart is required to enable or disable binary logging © 2014 26 IBM Corporation
  27. 27. Configuring binary logging (Liberty)  Binary log and trace settings are set in child elements of the logging element in the server.xml  Settings for the console log, trace specification, and log directory are inherited from the logging element <logging logDirectory="${server.output.dir}/logs" traceSpecification="*=info" consoleLogLevel="AUDIT"> <binaryLog purgeMaxSize="50"/> <binaryTrace purgeMaxSize="50"/> </logging> © 2014 27 IBM Corporation
  28. 28. Configuring HPEL (Full Profile) Administrators can independently configure (as an immediate runtime change or a persistent config change): • HPEL Log • HPEL Trace • HPEL Text log Log detail level setting is the same as previous releases. © 2014 28 IBM Corporation
  29. 29. Viewing logs and trace (Liberty )  The binaryLog command is in the wlp/bin directory  To view all binary logs and trace binaryLog view serverName  To view all warnings and errors binarylog view serverName --minLevel=WARNING  To view all warnings and errors from thread 33 from the latest run of the server binarylog view serverName –-includeThread=33 --includeInstance=”latest”  To tail the content of the logs and trace as they are written by the server binarylog view serverName --monitor  To see all of the available options binarylog help view © 2014 29 IBM Corporation
  30. 30. Viewing logs and trace (Full profile)  The logViewer command is in the profile bin directory  To view all binary logs and trace LogViewer | LogViewer -monitor © 2014 30 IBM Corporation
  31. 31. Viewing logs and trace (Full profile)  The logViewer command is in the profile bin directory  To view all warnings and errors LogViewer -minLevel WARNING © 2014 31 IBM Corporation
  32. 32. Viewing logs and trace (Full profile)  The logViewer command is in the profile bin directory  To view all binary logs and trace LogViewer -minLevel WARNING -latestInstance © 2014 32 IBM Corporation
  33. 33. Viewing logs and trace (Full profile)  The logViewer command is in the profile bin directory  To view all binary logs and trace LogViewer -startDate "05/31/12 22:10:40:000 EDT" -stopDate "05/31/12 22:10:58:000 EDT" -thread 91 -minLevel info © 2014 33 IBM Corporation
  34. 34. Viewing logs and trace (Full profile) Administrative Console 1. Expand Troubleshooting 2. Select Logs and trace 3. Select server1 4. Select View HPEL logs and trace © 2014 34 IBM Corporation
  35. 35. Copying log and trace (Liberty)  The binaryLog command lets you make new binary copies of your data  Any of the filtering options can be used when making a new copy  To create a new binary copy of your log and trace data with only messages that contain the string “hello” binarylog copy serverName myOutputDirectory --includeMessage="*hello*"  To view all log and trace data stored in this new copy binarylog view myOutputDirectory  To see all of the available options binarylog help copy © 2014 35 IBM Corporation
  36. 36. Copying log and trace (Full profile)  The logViewer command lets you make new binary copies of your data  Any of the filtering options can be used when making a new copy  To create a new binary copy of your log and trace data with only messages that contain the string “hello” logViewer -extractToNewRepository myOutputDirectory -message "*hello*"  To view all log and trace data stored in this new copy logviewer -repositoryDir myOutputDirectory  To see all of the available options logViewer -help © 2014 36 IBM Corporation
  37. 37. Extensibility  Developers can add key-value pairs to log and trace records using the LogRecordContext API – For example, in an application hosting multiple stores, you could add the storeId to each log and trace record in a servlet filter – Key-value pairs appear in log output when the advanced format is used (selectable when running the [Liberty] binaryLog / [Full Profile] logViewer command) – Key-value pairs can be used as filter parameters in [Liberty] binaryLog / [Full Profile] logViewer command – [Full Profile] appName extension is provided automatically, and other extensions are provided for thing like cross component trace, and SIP – Liberty: binarylog view serverName –-includeExtension=”storeId=WidgetsPlus” --format=advanced – Full Profile: logViewer -includeExtensions “appName=MyMall” ”storeId=WidgetsPlus” -format advanced [11/26/13 8:15:13:495 EST] 00000074 I UOW= source=com.mymall.ServletX class=com.mymall.Serv letX method=doGet org= prod= component= thread=[WebContainer : 0] storeId=[WidgetsPlus] Welcome to Widget's Plus! © 2014 37 IBM Corporation
  38. 38. Using High Performance Extensible Logging / Binary Logging with SplunkTM  Step 1 - Install Splunk app for WAS  Step 2 - Configure Splunk forwarder – Option #1 (simplest) - Read from TextLog files – Add monitor stanza to Splunk inputs.conf file to read from TextLog (use index=websphere and sourcetype=WebSphere:SystemOutErrLog) – Option #2 (any environment) - Read from logViewer -monitor script – Configure Splunk to read output from logViewer -monitor – Set up a no-arg script to run logViewer -monitor from profile bin – Add script stanza to Splunk inputs.conf file to run the above created script (use index=websphere sourcetype=WebSphere:SystemOutErrLog interval=-1) – Add -Xrs to Java command invocation in WASbinlogViewer script (for graceful shutdown when Splunk stops) © 2014 38 IBM Corporation
  39. 39. Using High Performance Extensible Logging / Binary Logging with SplunkTM  Option #1 (simplest) - Read from TextLog files inputs.conf [monitor://D:WASprofilesAppSrv01logsserver1] whitelist=Text.*.log$ crcSalt = <SOURCE> disabled = false followTail = 0 index = websphere sourcetype = WebSphere:SystemOutErrLog © 2014 39 IBM Corporation
  40. 40. Using High Performance Extensible Logging / Binary Logging with SplunkTM  Option #2 (any environment) - Read from logViewer -monitor script inputs.conf [script://$SPLUNK_HOMEetcsystembinrunLogViewer.cmd] disabled = false interval = -1 index = websphere sourcetype = WebSphere:SystemOutErrLog runLogViewer.cmd call D:WASprofilesAppSrv01binlogViewer -monitor WASbinlogViewer.cmd change: %JAVA_EXE% ... com.ibm.ws.logging.hpel.viewer.LogViewer %* to: %JAVA_EXE% -Xrs ... com.ibm.ws.logging.hpel.viewer.LogViewer %* © 2014 40 IBM Corporation
  41. 41. 41 Full Profile
  42. 42. Full Profile Problem Determination IBM Support Assistant Data Collector Cross Component Trace TimedOperations Feature Memory Leak Detection, Prevention, and Correction © 2014 42 IBM Corporation
  43. 43. IBM Support Assistant Data Collector  The isadc command is a console utility you can run from your profile bin directory (starting from WAS 8.0.0.6) to collect diagnostic information from your application server when you are experiencing problems. AppServerbin> isadc  The IBM Support Assistant data collector executes scripts which follow the same steps as detailed in MustGather documents produced by IBM L2 support teams for common problems.  Use of the IBM Support Assistant Data Collector can speed up problem determination by ensuring you have the right artifacts to work with support teams  The isadc command must be run on the system on which you are experiencing problems. Many of the scripts require the server to be running, and in the problem state.  The resultant data collected can be sent to IBM via secure file transfer, or kept for your own purposes. © 2014 43 IBM Corporation
  44. 44. IBM Support Assistant Data Collector  The IBM Support Assistant data collector is menu driven – you can choose from dozens of possible problem symptoms to gather problem-specific artifacts. Responses to menu choices and prompts can be recorded to a response file as follows: AppServerbin> isadc -record response.txt Response files are in plain text and can be reused for future invocations of the tool: AppServerbin> isadc response.txt  The tool provides a way to securely transfer data to IBM ECuRep, if desired. © 2014 44 IBM Corporation
  45. 45. IBM Support Assistant Data Collector Console (text) Interface isadc.bat or isadc.sh in WAS_HOME/bin Silent response capability Collection Archive Transfer to IBM for analysis with a PMR © 2014 45 IBM Corporation
  46. 46. Cross Component Trace  Cross Component Trace (XCT) is a log/trace correlation technology.  XCT enables you to determine which log/trace entries are part of each request.  XCT can be used in any of three different modes: 1)Request ID mode 2)Request ID and correlation log record mode 3)Request ID, correlation log record, and data snapshot mode  XCT works best in combination with High Performance Extensible Logging (HPEL). • HPEL stores XCT Request IDs • HPEL can filter log/trace records by XCT Request ID © 2014 4466 IBM Corporation
  47. 47. Cross Component Trace Logviewer  IBM WebSphere Cross Component Trace Logviewer can be used to view files augmented with correlation log records. • Available for the IBM Support Assistant • Can load multiple files simultaneously • Can show flat / hierarchical views © 2014 4477 IBM Corporation
  48. 48. Enabling XCT through Administrative Console  The panel below is where XCT is enabled on the administrative console • The panel can be found here: WebSphere Application Servers > SERVER_NAME > Change log detail levels © 2014 4488 IBM Corporation
  49. 49. Understanding XCT Correlation Log Records  Correlation log records look like this example: [4/23/12 13:54:44:509 IST] 0000008e XCT I BEGIN AAADx/itMDz-AAAAAAAAAAA 00000000000-cccccccccc2 HTTPCF(InboundRequest /JMSApp/LocalMessageSend RemoteAddress(127.0.0.1) RequestContext(2082603117))  <Date>: The date and time when the log record was generated  <Thread_ID>: The thread which generated this message  <XCT_Logger_Name>: The XCT logger name is XCT. This logger is used to identify the XCT Records in the Log file  <Message_Type>: Type of the log message  <XCT_STATE>: Each XCT Record has a State, it can be BEGIN, END  <XCT_ID>: A Unique ID generated for correlating the XCT Records  <XCT_PARENT_ID>: The XCT_ID of the parent XCT context  <XCT_MESSAGE>: The XCT message contains the information about the XCT record; this can contain some Associations and Annotations © 2014 4499 IBM Corporation
  50. 50. Identifying all log/trace entries that are part of the same HTTP request  The XCT requestID is added to all log and trace records associated with HTTP requests.  The requestID can only be seen when using the HPEL logViewer command-line tool with the advanced format logViewer -thread 91 -minlevel info -format advanced © 2014 5500 IBM Corporation
  51. 51. Identifying all log/trace entries that are part of the same HTTP request  The XCT requestID is added to all log and trace records associated with HTTP requests. – The requestID can only be seen when using the HPEL logViewer command-line tool with the advanced format logViewer -includeExtensions requestID=AAAKLwUVkuH-AAAAAAAAAAC © 2014 5511 IBM Corporation
  52. 52. IBM WebSphere Cross Component Trace Logviewer  The scenarios following this slide use the IBM WebSphere Cross Component Trace Logviewer – available as a tool add-on for the IBM Support Assistant  Tool used to examine XCT entries in a log  Logs can be loaded from multiple servers and they are stitched together © 2014 5522 IBM Corporation
  53. 53. Memory leak detection, prevention and correction  Customers have discovered several Classloader and ThreadLocal leaks in WebSphere Application Server and their own applications  Prior to this feature, WebSphere Application Server does not contain an application level that provides top down memory leak detection and protection  Customers want to increase application uptime without cycling the server  Frequent redeployments of the application result in OOM errors Existing (v7 & up) PM39870: Improved classloader leak detection. NEW in V8.5 (Not in Liberty Profile)  Prevention – Code to proactively fix suspect application classloader leak patterns  Detection - Recognize Application triggered classloader leaks & provide diagnostics  Fixing – Leverage existing JDK APIs and reflection to remedy classloader leaks.  Enabled by setting the setting JVM custom properties © 2014 5533 IBM Corporation
  54. 54. Class Unloading 101 A Java Object has a reference to its Class object.getClass() A Class has a reference to its ClassLoader class.getClassloader() A ClassLoader has a reference to every Class it has loaded classloader.findLoadedClass() Java classes are loaded per-class, but unloaded per-classloader Class unloading can be denied because of references to ClassLoader, Class or Object – References can come from anywhere: • Other Objects • Other Llasses/ClassLoaders • Thread stacks • Thread variables • JNI global references • Finalizer queue entries © 2014 5544 IBM Corporation
  55. 55. Unwanted reference to application classloader – example 1 ■ Here, we have a CompoundClassLoader which is kept alive because a thread named “Keep- Alive-Timer” has its contextClassLoader set to it ■ “Keep-Alive-Timer” is a daemon thread – Spawned by the classlibraries – Daemon threads live until the JVM ends ■ Threads inherit contextclassloader – From their parent ■ This is a Java classlibrary bug – Being raised with Oracle – Fix is simple: • thread.setContextClassLoader(null); 55 © 2010 IBM Corporation
  56. 56. Unwanted reference to a class which was loaded by the application classloader – example 2 ■ Here, a CompoundClassLoader is kept alive because a class it loaded – org.richfaces.model.selection.ClientSelecti on – has been stored inside a HashMap in the system class java.beans.PropertyEditorManager ■ Because this is a system class, it has javadoc! ■ Looks like a RichFaces bug... – https://jira.jboss.org/browse/RF-7911 • “OutOfMemory when redeploying - ClientSelection not unregistered from PropertyEditorManager” 56 © 2010 IBM Corporation
  57. 57. Common leaks ■ ThreadLocal problems – Custom class extending ThreadLocal? ■ Threads' contextClassLoaders – Daemon threads started by a servlet – Careless use of java.util.Timer – Daemon threads started by 3rd party libraries, shared between two applications ■ Bean introspection – If you introspect (call getBeanInfo()) on a Bean loaded by the app classloader, you must call flushfromCaches(beanClass) on app shutdown ■ JMX MBeans and NotificationListeners – Must be unregistered when the application stops 57 © 2010 IBM Corporation
  58. 58. Example BAD code public class MyCounter { private int count = 0; public void increment() { count++; } public int getCount() { return count; } } public class MyThreadLocal extends ThreadLocal<MyCounter> { } public class LeakingServlet extends HttpServlet { private static MyThreadLocal myThreadLocal = new MyThreadLocal(); protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { MyCounter counter = myThreadLocal.get(); if (counter == null) { counter = new MyCounter(); myThreadLocal.set(counter); } response.getWriter().println( "The current thread served this servlet " + counter.getCount() + " times"); counter.increment(); } } 58 © 2010 IBM Corporation
  59. 59. Example BAD code public class MyCounter { private int count = 0; public void increment() { count++; } public int getCount() { return count; } } public class MyThreadLocal extends ThreadLocal<MyCounter> { } public class LeakingServlet extends HttpServlet { private static MyThreadLocal myThreadLocal = new MyThreadLocal(); protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { MyCounter counter = myThreadLocal.get(); if (counter == null) { counter = new MyCounter(); myThreadLocal.set(counter); } response.getWriter().println( "The current thread served this servlet " + counter.getCount() + " times"); counter.increment(); } } 59 © 2010 IBM Corporation
  60. 60. Detection, prevention and action  Detection: Issue warnings when a memory leak is detected  Prevention is on by default and applies only to JRE triggered leaks.  Action: Take proactive action to fix memory leaks.  Actions have reasonable defaults and are configured on a case-by-case  Pseudo code for clearing leaks protected void com.ibm.ws.classloader.clearReferences() { if(ENABLE_CLEAR_REFERENCES_JDBC) clearReferencesJdbc(); if(ENABLE_CLEAR_REFERENCES_THREADS) clearReferencesThreads(); if(ENABLE_CLEAR_REFERENCES_THREADLOCALS) clearReferencesThreadLocals(); if(ENABLE_CLEAR_REFERENCES_RMI_TARGETS) clearReferencesRmiTargets(); if(ENABLE_CLEAR_REFERENCES_STATICS) clearReferencesStaticFinal(); } © 2014 60 IBM Corporation
  61. 61. Leak detection messages  CWMML0015E: The web application [WasSwat#WasSwatWeb.war] created a ThreadLocal with key of type [test.memleak.MyThreadLocal] (value [test.memleak.MyThreadLocal@216c691]) and a value of type [test.memleak.MyCounter] (value [test.memleak.MyCounter@21942ff]) but failed to remove it when the web application was stopped.  CWMML0010E: The web application [LeakApp#leak.war] appears to have started a thread named [Thread-73] but has failed to stop it.  CWMML0011E: The web application [LeakApp#leak.war] appears to have started a TimerThread named [leaked-thread] via the java.util.Timer API but has failed to stop it.  CWMML0024W: About to interrupt thread [leakingThread] which is currently executing  CWMML0026I: ClassLoader memory leak is fixed. Clearing leak References succeeded for LeakApp#leak.war. © 2014 61 IBM Corporation
  62. 62. Useful Links – Runtime Capabilities High Performance Extensible Logging Using High Performance Extensible Logging to troubleshoot applications http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/index.jsp? topic=%2Fcom.ibm.websphere.nd.multiplatform.doc %2Fae%2Fttrb_usinghpel.html High Performance Extensible Logging (HPEL) http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/i ndex.jsp? topic=/com.ibm.iea.was_v8/was/8.0/ProblemDeterminatio n/WASv8_HPEL/player.html System administration in WebSphere Application Server V8.5, Part 3: High Performance Extensible Logging (HPEL) http://www.ibm.com/developerworks/websphere/techjourn al/1208_bourne/1208_bourne.html Cross Component Trace WebSphere Application Server v8.5 Cross Component Trace http://webspherecommunity.blogspot.ca/2012/07/websph ere-application-server-v85-cross.html Use Cross Component Trace (XCT) request Ids to see which log entries were generated by each request http://www.youtube.com/watch?v=oXsfDoiNb5c Use the IBM WebSphere Cross Component Trace Logviewer to view WebSphere Application Server log files http://www.youtube.com/watch?v=qorwRZh4DgQ High Performance Extensible Logging and Cross Component Trace Tech Video https://www.youtube.com/watch?v=FKKfUdNux70 IBM Training: IBM WebSphere Application Server V8 Problem Determination http://www- 304.ibm.com/jct03001c/services/learning/ites.wss/us/en? pageType=course_description&courseCode=WU582 © 2014 62 IBM Corporation
  63. 63. Useful Links – Runtime Capabilities Memory Leak Detection and Prevention Configuring the memory leak policy http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/topic/com.i bm.websphere.nd.doc/ae/ttrb_configmemleak.html Memory leaks in Java Platform, Enterprise Edition applications http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/index.jsp? topic=%2Fcom.ibm.websphere.nd.doc%2Fae %2Fctrb_memleakdetection.html Binary Logging Liberty Profile: Binary Logging http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/topic/com.i bm.websphere.wlp.express.doc/ae/cwlp_HPELOverview. html Binary logging for Liberty profile http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/t opic/com.ibm.iea.was_v8/was/8.5.5.0/content/WASV855_ Binary_Logging/player.html? dmuid=20130807101913462846 © 2014 63 IBM Corporation
  64. 64. Useful Links – Runtime Capabilities Timed Operations Timed operations for Liberty http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/t opic/com.ibm.iea.was_v8/was/8.5.5.0/content/WASV855_ Serviceability_TimedOperations/player.html? dmuid=20130815082326115285 Lab: Liberty profile timed operations for JDBC http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/t opic/com.ibm.iea.was_v8/was/8.5.5.0/content/WASv855_ Liberty_Timed_Operations_Lab.pdf? dmuid=20130807102148029081 Files for lab: http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/topic/com.Sensitive Log and Trace Guard Using sensitive log and trace guard http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/topic/com.i bm.websphere.express.doc/ae/ctrb_sensitivelogtrace.html © 2014 64 IBM Corporation
  65. 65. IBM Support Assistant v5.0 65
  66. 66. IBM Support Assistant 5.0 ISA 5.0 Team Server •Server-based model •Install once - shared by many team members via browser •Web 2.0 browser interface •Remote execution of PD tools •Off-load analysis processing •Collaboration on PD •Case Management •Tool Management •Single-user option available Browser Browser Browser Browser © 2014 66 IBM Corporation Available at www.ibm.com/software/support/isa ISA Team Server
  67. 67. IBM Support Assistant 5.0 – Deployment options Team Server Single install Multiple end users Leverages resources of ISA server system Shared investigation Standalone Single user Local install User administered ibm.com (ISA and Tool updates) ISA Team Server ibm.com (ISA and Tool updates) ISA Team Server (local) © 2014 67 IBM Corporation
  68. 68. Automation of Data Upload to ISA 5 ■ ISA Provide a REST interface for some of its functions: ■ Case creation: curl --user user1:user1password -X POST -H Content-Type:application/x-www-form-urlencoded -v -o /home/dumps/curl.log --url http://myISA5TeamServer.mydomain.net/rest/1/tickets -d "summary={case summary}&description={case description}" ■ File upload: curl --user user1:user1password -X POST -v -k -o /home/dumps/curl.log -F file=@'{}' --url http://myISA5TeamServer.mydomain.net/rest/1/files/${caseno}/file_upload?path=/${caseno}/ ■ Can be combined with IBM JDKs “-Xdump” options to run automatically: -Xdump:tool:events=systhrow,filter=java/lang/OutOfMemoryError,exec="DumpUpload.sh %pid” 68 © 2010 IBM Corporation
  69. 69. Tools – Toolbox •Catalog of available tools •Learn about and discover tools •Launch •Tool Help •Execution History © 2014 6699 IBM Corporation
  70. 70. Files – Launch tools  Suggested tools based on file type  Automatically pass files to a tool  All tools available from the “Other…” menu © 2014 7700 IBM Corporation
  71. 71. Available Problem Determination Tools in ISA 5.0 Team Server Tool Name Problem Areas Type Classloader Analyzer Java™, Runtime Desktop Database Connection Pool Analyzer WebSphere, Runtime Desktop FileNet Optical Storage And Retrieval (OSAR) Cable Tool Desktop Garbage Collection and Memory Visualizer (GCMV) Java, Performance, Memory Desktop, Report Health Center Java, Performance Desktop HeapAnalyzer Java, Memory Desktop Interactive Diagnostic Data Explorer (IDDE) Java, Memory Desktop+Server/Web Memory Analyzer (MAT) Java, Memory Desktop, Report, Web Pattern Modeling and Analysis Tool (PMAT) Java, Performance, Memory Desktop, Report Portal Log Analyzer WebSphere, Configuration Report Processor Time Analysis Tool for Linux Java, Performance Desktop Profile Port Checker WebSphere, Configuation Report Thread and Monitor Dump Analyzer (TMDA) Java, Runtime Desktop, Report Trace and Request Analyzer for WebSphere Application Server WebSphere, Runtime Desktop Web Server Plug-in Analyzer for WebSphere Application Server WebSphere, Perfomance Desktop WebSphere Application Server Configuration Visualizer WebSphere, Configuration Report WebSphere Cross Component Trace Logviewer WebSphere, Runtime Desktop © 2014 7711 IBM Corporation
  72. 72. Additional Resources ISA YouTube playlist @ISA_Tools Requests for Enhancement Contact Information Need support or have questions about Team Server? Visit our forum: http://ibm.biz/ISA-Forum IBM Support Assistant web page http://www.ibm.com/software/support/isa © 2014 72 IBM Corporation
  73. 73. Useful Links – Problem Determination Tools IBM Support Assistant IBM Support Assistant http://www.ibm.com/software/support/isa/ WAIT IBM Whole-system Analysis of Idle Time (WAIT) http://wait.ibm.com © 2014 73 IBM Corporation
  74. 74. Useful Links – Knowledge and Education IBM Training IBM WebSphere Application Server V8.5.5 Problem Determination http://www- 304.ibm.com/jct03001c/services/learning/ites.wss/ca/en? pageType=course_description&courseCode=WA591CE WebSphere Support Technical Exchanges Support Technical Exchanges http://www.ibm.com/support/entry/portal/Scheduled_tech_ exchanges/Software/WebSphere/WebSphere_brand_sup port_(general) IBM Education Assistant IBM Education Assistant http://www.ibm.com/software/info/education/assistant/ Problem Determination Practices RedPaper – Increasing Resiliency for IBM WebSphere Application Server Deployments http://www.redbooks.ibm.com/redpapers/pdfs/redp5033.p df © 2014 74 IBM Corporation
  75. 75. 75 Questions?
  76. 76. Your feedback is valuable - please complete your session or lab evaluation! Session number [A6] © 2014 IBM Corporation Provide your evaluations by: Evaluation forms: Fill out a form at the end of each session Paper forms are located in each of the session or lab rooms Place the completed form in the tray as you exit the room - Or – Complete the session survey on Event Connect Portal: ibmeventconnect.eu/euxdx Select Sessions, then Session Finder, and complete the survey
  77. 77. Visit WASdev.net for downloads and other resources WASdev.net
  78. 78. For Additional Information © 2014 IBM Corporation  IBM Training http://www.ibm.com/training  IBM WebSphere http://www-01.ibm.com/software/be/websphere/  IBM developerWorks www.ibm.com/developerworks/websphere/websphere2.html  WebSphere forums and community www.ibm.com/developerworks/websphere/community/ 78

×