Web sphere application server performance tuning workshop

3,074 views

Published on

WebSphere performance tuning toolkit workshop

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,074
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
298
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Web sphere application server performance tuning workshop

  1. 1. WebSphere Application Server Performance Tuning Toolkit Workshop with DaytraderStatement: draft
  2. 2. Author : Sui Peng Fei (suipf@cn.ibm.com)Version: 1.0Date: Jan, 16th, 2012
  3. 3. Table of Contents Getting Start.....................................................................................................................................4 Performance Tuning Toolkit Overview.......................................................................................................4 Prepare for the workshop..........................................................................................................................6 Set up WebSphere Application Server Topology........................................................................................6 Download the WebSphere Application Server Performance Tuning Toolkit(PTT)......................................6 Download the Apache JMeter...................................................................................................................6 Install the Daytrader Application...............................................................................................................6 Have a brief stress testing using the JMeter Tool.....................................................................................10 Scenario1: “Synchronization Blocking” Problem............................................................................13 Determine the “Synchronization Blocking” problem...............................................................................16 Generate a thread dump.........................................................................................................................16 Launch the ISA 5.0 Tech Preview .............................................................................................................17 Use ISA V5 to analyze the “Synchronization Blocking” problem..............................................................18 Scenario2: “Dead Lock” problem..................................................................................................22 Monitoring the “Dead Lock” problem......................................................................................................23 Determine the “Dead Lock” problem.......................................................................................................25 Generate a thread dump.........................................................................................................................25 Use ISA V5 to analyze the “Dead Lock” problem......................................................................................26 Scenario3: “High CPU Usage” problem.........................................................................................30 Trigger the “High CPU Usage” problem....................................................................................................30 Monitoring the “High CPU Usage” problem.............................................................................................31 Determine the “High CPU Usage” problem..............................................................................................33 Scenario4: “Connection Leak” problem........................................................................................35 Application Changes to Simulate “Connection Leak” Situation................................................................35 Trigger the “Connection Leak” problem...................................................................................................36 Monitoring the “Connection Leak” problem............................................................................................37 Determine the “Connection Leak” problem.............................................................................................39 Scenario5: “Memory Leak” problem.............................................................................................42 Application Changes to Simulate “Memory Leak” Situation....................................................................42 Trigger the “Memory Leak” problem.......................................................................................................42 Monitoring the “Memory Leak” problem................................................................................................43 Determine the “Memory Leak” problem.................................................................................................44 Other optional functions you can go through................................................................................49 Tuning the performance related parameters...........................................................................................49 Adjust rules from rule editor if needed....................................................................................................50 Generate report.......................................................................................................................................52
  4. 4. Getting StartPerformance Tuning Toolkit OverviewThe WebSphere® Application Server Performance Tuning Toolkit (PTT) is anEclipse-based intelligent tool designed to help users tune the performance ofWebSphere Application Server using data analysis, and statistical inferencetechnology. It is designed to help users locate bottlenecks and tune their applicationsappropriately.Compared with other performance tuning tools, the main character of PTT is lightweight and easy to use. You need not do any install or configuration work exceptproviding the IP and soap port of your Dmgr.Below is the function list of the toolkit:Functions ComponentsPerformance  Dashboard (key status of all servers)Monitoring  Monitor page (chart of components of one server)  Data page (detailed page of components)Performance Tuning  “Tuning Parameters” view (tune common server parameters)  Scripts view (tune other parameters)Health Check  Rule engine (process data)  Rule editor (edit rules)Operations  Dump agent (generate thread dump and heap dump)  Trace agent (enable trace)  Connection agent (extract the connection pool contents)Generate Report  Report engine (generate, export and print report)In order to help users learn how to use this toolkit, it provides a modified Daytraderbenchmark and a set of scripts to illustrate existing common customer issues andsolutions, thus greatly reducing the user’s learning curve. Through execution of theworkshop, you will have a good understanding of the impact of typical bad code, and
  5. 5. know more about symptoms of certain problems thereby speeding the process ofproblem resolution.The built-in Daytrader applicationDayTrader is a open source benchmark built on a core set of Java EE technologiesthat includes Java Servlets and JavaServer Pages (JSPs) for the presentation layer andJava database connectivity (JDBC), Java Message Service (JMS), EnterpriseJavaBeans (EJBs) and Message-Driven Beans (MDBs) for the back-end businesslogic and persistence layer. The following diagram provides a high-level overview ofthe application architecture. Figure 1: Application architectureThe built-in Daytrader application will simulate improper application design or someerror coding scenario which will involve "Performance" issues.The simulated "sick" situations include: 1. Causal factors lead to synchronization blocking or lock competition, and the threads have to spend more time on waiting for the lock. 2. Causal factors lead to a deadlock and block other threads which need to gain the lock owned by deadlock threads. 3. Causal factors lead to heavy JDBC connection consuming, which will cause lots of connection waiting threads. 4. High CPU usage cause by some CPU consuming threads. 5. Out of memory problems caused by memory leak.
  6. 6. Prepare for the workshop Set up WebSphere Application Server TopologyInstall the WebSphere Application Server 7.0 or later and create a cluster with at least2 members (Download the trial version from DeveloperWorks if needed). Create aproxy server as the dispatcher. Download the WebSphere Application Server Performance Tuning Toolkit(PTT)Download link (Internal):http://cattail.boulder.ibm.com/cattail/#view=suipf@cn.ibm.com/files/2F87C2500E943DDC86A2F0EC093F23B6 Download the Apache JMeterDownload link: http://jmeter.apache.org/download_jmeter.cgiDownload the ISA 5.0 Tech Preview from IBM website.Download link: http://www-01.ibm.com/support/docview.wss?uid=swg27023689Install the Daytrader ApplicationUnzip the PTT zip file and double click the “PerfTuningToolkit.exe” to launch thePTT workbench
  7. 7. Figure 2: PTT workbenchClick the “Add host” button in the toolbar of “Host” view to add a new host. Justinput the IP address and soap port of Dmgr. You need also provide the user name andpassword if the global security of WAS is enabled. Figure 3: Add the host infoClick the “Help” -> “Case Study” from the workbench menu bar to import the sampleapplication and script files to the “scripts” view.
  8. 8. Open the settings.py in the “Sample” folder and change the cluster name to the clusteryou created in previous steps. You also need to set the security settings if the WASsecurity is enable. Figure 4: Import the assets and modify settingsSelect the host of your Dmgr in the host view, and right click the installDaytrader.pyin the sample folder of the “scripts” view. Select “run script” menu in the contextmenu to run the installDaytrader.py script against your cell. The script will install themodified Daytrader Application to the cluster you specified.
  9. 9. Figure 5: Install the Daytrader Application Figure 6: Check the results of installationImportant: (Re)start the cluster and proxy server, and manually check the installedapplication from url http://<proxy server host name>:<port>/daytrader/scenario
  10. 10. Figure 7: Verify the installation of Daytrader applicationNote: the Daytrader application will use an embedded Derby data base, it is not avalid design for a normal clustered application, but is very suitable for a demopurpose.You can also use DB2 as a central db for performance related testing, but you need tospecified the db2 settings in the settings.py and also need to created table andpopulate data according to the instruction on the “configuration” tab of Daytraderapplications.Have a brief stress testing using the JMeter ToolLaunch the JMeter tool and open the daytrader.jmx file in the<PTThome>/workspace/Scripts/sample folder.Change the hostname and port number (default is 80) to the host name and port
  11. 11. number of your proxy server. Figure 8: Open the daytrader.jmx and modify the host name and port Figure 9: Start the stressDouble click the host in the “Hosts” view of PTT workbench to connect to the celland start monitoring. It is recommended monitoring at least 10 minutes to make sure
  12. 12. no error will occur during normal stress testing. Figure 10: Monitor your cellNote: The dashboard will monitor all the key status of servers (include ApplicationServers and Proxy Servers) in the latest monitoring interval (monitoring interval ismultiple of the data collection interval which was set in the preference panel)The color of performance data will turn red if some abnormal events were detected incurrent monitoring interval, it only represent the current status. There will also be awarning mark in the second column as long as any error ever occurs and it will leaveunless you clear it. So the warning marks can represent history abnormal events. Figure 10: Errors in the dashboard
  13. 13. Scenario1: “Synchronization Blocking” ProblemApplication Changes to Simulate “Synchronization Blocking”SituationIn this scenario, the application will continuously call static synchronized method, inwhich the thread will sleep several seconds.Sample code changes in the TradeAppServlet//For Synchronization Blockingif(TradeConfig.isThreadHang()){ syncMethod(); }public static synchronized void syncMethod(){ try { Thread.sleep(TradeConfig.getHangTime()); } catch (InterruptedException e) { e.printStackTrace(); } }Trigger the “Synchronization Blocking” problemImportant: make sure that the stress is running and the PTT is monitoring before youtrigger the “Synchronization Blocking” problemClick the “Server” tab of the home page (http://hostname:port/daytrader). The errorreproduction options will be shown (see figure11). In order to trigger thesynchronization blocking problems, select the “Synchronization Blocking” checkboxin the “Server:” tab and update the runtime configuration. If you find that the hangtime is not long enough to trigger the thread hung problem, you can increase thisvalue accordingly..
  14. 14. Figure11. Error reproduction optionsMonitor the “Synchronization Blocking” problemWhen a synchronization blocking problem occurs, you will see the red data andwarning marks in the dashboard as below. Figure12. Dashboard monitoring
  15. 15. Double click the server panel to switch to the monitoring page. In the servlet responsetime section, you can see that the average response time keeps increasing. Figure13. Server chart monitoringClick the point to switch to the data page to check the servlet that is blocking. Figure14. Check the detailed dataBesides, some servlet alerts will appear in the alert section. Figure15. The alert panelClick the alert point to switch to the data page, which show that the time limit ofservlet average response time is exceeded.
  16. 16. Figure 16: Alert monitoring dataDetermine the “Synchronization Blocking” problem Generate a thread dumpIn order to determine the “Synchronization Blocking” problem, we need to generate a“Thread Dump” of the sick server from the content menu of the “Topology” view.
  17. 17. Figure 17: Take a thread dump of the sick server Launch the ISA 5.0 Tech PreviewPerform the following steps to launch ISA 5.0 Tech Preview.1) Unzip the zip file to your file system.2) Start WAS CE server with ISA5/start_isa.bat3) Access ISA 5.0 with URL: http://localhost:8080/isa5/
  18. 18. Figure 18: GUI of ISA 5.0 Tech Preview Use ISA V5 to analyze the “Synchronization Blocking” problemAdd a new test case for the “Synchronization Blocking” problem. Figure 19: Add new case
  19. 19. Figure 20: Input case infoAdd the thread dump files generated to the test case. By default, they are generated inthe profile root folder of the sick server. Figure 21: Add thread dump to the caseNext, analyze the thread dump file using the “Thread and Monitor Dump Analyzer”tool of ISA v5 and view the analysis results.
  20. 20. Figure 22: Analyze the thread dump Figure 23: Analysis finished Figure 24: View the analysis resultsYou need to check the detail info of monitors and threads to find the root cause.
  21. 21. Figure 21: Analysis results summary Figure 22: “Monitor Dump” analysis summary Figure 23: “Monitor Dump” analysis detailed infoFrom the detailed page we can find the blocked threads and the status of lock owner,so we can improve our application accordingly.
  22. 22. Scenario2: “Dead Lock” problemApplication Changes to Simulate “Dead Lock” SituationIn this scenario, a dead lock problem will be triggered and all the other threads will behung for ever, as they can not get the synchronization lock which is owned by thedeadlock threads.Below is the code changes in the TradeAppServlet//For Dead Lockif(TradeConfig.isDeadLock()){ deadLockMethod(); }public void deadLockMethod(){ try { synchronized (lock1) { // lock1 is defined in the "Methodsand Static Variables" tab Thread.sleep(5000); synchronized (lock2) { } } synchronized (lock2) { // lock2 is defined in the "Methodsand Static Variables" tab Thread.sleep(5000); synchronized (lock1) { } } } catch (Exception e) { e.printStackTrace(); } }Trigger the “Dead Lock” problemImportant: make sure that the stress is running and the PTT is monitoring before youtrigger the “Dead Lock” problem
  23. 23. In the “Server” tab of Daytrader home page, select the “Deadlock” checkbox andclick the “Update Configuration” button. A deadlock will be triggered. It will not onlyblock the deadlock lock threads, but also block other threads which have to wait forthe lock owned by the deadlock threads. Figure24: Error reproduction optionsMonitoring the “Dead Lock” problemWhen a dead lock problem occurs, you will see the red data and warning marks in thedashboard.
  24. 24. Figure25: Dashboard monitoringDouble click the server panel to switch to the monitoring page, and you can see thatsome servlet alerts are shown in the alert section. Figure26: Server monitoringClick the alert point in the chart to check the alert content in the detailed data page.
  25. 25. Figure27: Alert detailed contentDetermine the “Dead Lock” problem Generate a thread dumpIn order to determine the “Dead Lock” problem, we need to generate a “ThreadDump” of the sick server from the content menu of the “Topology” view.
  26. 26. Figure 28: Take a thread dump of the sick server Use ISA V5 to analyze the “Dead Lock” problemAdd a new test case for the “Dead Lock” problem.
  27. 27. Figure 29: Add new caseFigure 30: Add thread dump to the case
  28. 28. Figure 31: Analyze the thread dump Figure 32: View the analysis resultsFigure 33: “Monitor Dump” analysis summaryFigure 34: “Monitor Dump” analysis summary
  29. 29. Figure 35: “Monitor Dump” analysis detailed infoFrom the detailed page we can find the deadlock threads and their status, so we canfix the problem accordingly. A best practice on this kind of problem is getting thelocks in the same order. So here is the improved code:public void deadLockMethod(){ try { synchronized (lock1) { // lock1 is defined in the "Methodsand Static Variables" tab Thread.sleep(5000); synchronized (lock2) { // lock2 is defined in the"Methods and Static Variables" tab } } synchronized (lock1) { Thread.sleep(5000); synchronized (lock2) { } } } catch (Exception e) { e.printStackTrace(); } }
  30. 30. Scenario3: “High CPU Usage” problemApplication Changes to Simulate “High CPU Usage” SituationIn this scenario, a CPUConsumer thread will be started which will invoke an emptyloop method. You can increase the thread number if the CPU usage is not highenough.Sample code changes in the TradeAppServlet//For High CPU Usage if(TradeConfig.isHighCpuUsage()){ if(cpuThreadCount<TradeConfig.getThreadNum()){ try { cpuThreadCount++; CPUConsumer cc = new CPUConsumer(); cc.start(); } catch (Exception e) { e.printStackTrace(); } } } class CPUConsumer extends Thread{ public void run() { try{ while(TradeConfig.isHighCpuUsage()){ } cpuThreadCount--; }catch(Exception e){ e.printStackTrace(); } }}Trigger the “High CPU Usage” problemIn the “Server” tab of Daytrader home page, select the “High CPU Usage” checkboxand click the “Update Configuration” button. A loop thread will be started to consume
  31. 31. the CPU resources. If you find that the CPU usage is not high enough to trigger the“HighCPU” health policy, you can increase the thread number accordingly Figure36: Error reproduction optionsMonitoring the “High CPU Usage” problemWhen a high CPU usage problem occurs, you will see the red data and warning marksin the dashboard.
  32. 32. Figure37: Dashboard monitoringDouble click the server panel to switch to the monitoring page, and you can see thatthe CPU usage is very high. Figure38: Server CPU monitoringSome runtime alerts are shown in the alert section. Figure39: Alert panelClick the alert point to check the alert content in the detailed data page
  33. 33. Figure39: Alert detailed contentDetermine the “High CPU Usage” problemIn order to determine the High CPU Usage problem, we need to collect the CPUusage info from the command line. But the command differs from OS to OS; here wewill take the Linux for example. Change the node name and server name of the sickserver and run the shell file with the following content and content.node="node1"server="TradeDC_node1"echo Start get cpuInof on `date` >> cpuInfo.txtecho==================================================================================================== >> cpuInfo.txtpid=`ps -ef |grep java | grep com.ibm.ws.runtime.WsServer | grep -w "$node $server" |awk {print $2}`echo "$pid" >> cpuInfo.txtif [ -n "$pid" ]then
  34. 34. top -b -n1 -H -p $pid >> cpuInfo.txt kill -3 $pidfiecho Finish custom action on `date` >> cpuInfo.txtOpen the cpuInfo.txt file generated by the “collectCPUInfo” custom action. You willfind the system CPU info as below:Start custom action on Mon Feb 20 23:20:01 CST 2012==================================================================================================32094top - 18:20:20 up 6 days, 8:25, 4 users, load average: 1.48, 1.38, 1.31Tasks: 180 total, 1 running, 179 sleeping, 0 stopped, 0 zombieCpu(s): 6.6%us, 0.5%sy, 0.0%ni, 92.4%id, 0.1%wa, 0.0%hi, 0.4%si, 0.0%stMem: 4152360k total, 4029544k used, 122816k free, 162536k buffersSwap: 2064376k total, 0k used, 2064376k free, 2350792k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 623 root 20 0 915m 494m 58m R 82.8 12.2 123:00.06 java32255 root 20 0 915m 494m 58m S 2.0 12.2 0:14.72 java32094 root 20 0 915m 494m 58m S 0.0 12.2 0:00.00 java32096 root 20 0 915m 494m 58m S 0.0 12.2 0:18.93 java32097 root 20 0 915m 494m 58m S 0.0 12.2 0:02.77 java32098 root 20 0 915m 494m 58m S 0.0 12.2 0:26.89 java32101 root 20 0 915m 494m 58m S 0.0 12.2 0:24.46 java32102 root 20 0 915m 494m 58m S 0.0 12.2 0:00.00 java32103 root 20 0 915m 494m 58m S 0.0 12.2 0:00.00 java32104 root 20 0 915m 494m 58m S 0.0 12.2 0:00.00 java32106 root 20 0 915m 494m 58m S 0.0 12.2 0:00.06 java32107 root 20 0 915m 494m 58m S 0.0 12.2 0:00.06 java32109 root 20 0 915m 494m 58m S 0.0 12.2 0:00.48 java32115 root 20 0 915m 494m 58m S 0.0 12.2 0:00.00 java32116 root 20 0 915m 494m 58m S 0.0 12.2 0:00.38 java32118 root 20 0 915m 494m 58m S 0.0 12.2 0:00.02 java32121 root 20 0 915m 494m 58m S 0.0 12.2 0:02.24 java List 1: Content of cpuInfo.txtFrom these info, we can find that the thread “623” consume 82.8% of CPU resource,which is the root cause of high CPU usage problem.As the thread dump will use “Hexadecimal” as the thread id, we need to convert thethread pid from 623 (Decimal) to 0x26F (Hexadecimal) firstly, and then search it inthe thread dump file, you will find the root cause is the CPUConsumer.run method
  35. 35. in TradeAppServlet. Below is the stack trace:3XMTHREADINFO "Thread-95" J9VMThread:0x0C0EFE00, j9thread_t:0x8D15AFA8, java/lang/Thread:0x9B2C2FC0, state:CW,prio=53XMTHREADINFO1 (native thread ID:0x26F, native priority:0x5, native policy:UNKNOWN)3XMTHREADINFO2 (native stack address range from:0x02B87000, to:0x02BC8000, size:0x41000)3XMTHREADINFO3 Java callstack:4XESTACKTRACE atorg/apache/geronimo/samples/daytrader/web/TradeAppServlet$CPUConsumer.run(TradeAppServlet.java:120)3XMTHREADINFO3 Native callstack:4XENATIVESTACK (0x00CA1752 [libj9prt24.so+0xb752])4XENATIVESTACK (0x00CACF60 [libj9prt24.so+0x16f60])4XENATIVESTACK (0x00CA17E5 [libj9prt24.so+0xb7e5])4XENATIVESTACK (0x00CA1908 [libj9prt24.so+0xb908])4XENATIVESTACK (0x00CA1584 [libj9prt24.so+0xb584])4XENATIVESTACK (0x00CACF60 [libj9prt24.so+0x16f60])4XENATIVESTACK (0x00CA15F8 [libj9prt24.so+0xb5f8])4XENATIVESTACK (0x00C9D500 [libj9prt24.so+0x7500])4XENATIVESTACK (0x0099740C) List 2: Content of thread dumpFrom the content of cpuInfo.txt and the thread dumps, we can find the threadsconsuming lots of CPU resources and their stack trace, so we will be able to fix theproblem accordingly.Scenario4: “Connection Leak” problemApplication Changes to Simulate “Connection Leak” SituationIn this scenario, some requests will invoke a ConnectionConsumer thread in which itwill get a connection from the connection pool and never close it. You can adjust the“Leak Frequency” to control the leak rate.Code changes in the TradeAppServlet//For Connection Consuming if(TradeConfig.isConnectionLeak()){ if(triggerByFrequency(TradeConfig.getLeakFrequency())){ ConnectionConsumer cc = new ConnectionConsumer(); cc.start(); }
  36. 36. } class ConnectionConsumer extends Thread{ public void run() { try{ javax.naming.InitialContext ctx = newjavax.naming.InitialContext(); javax.sql.DataSource ds = (javax.sql.DataSource)ctx.lookup("jdbc/TradeDataSource"); java.sql.Connection con = ds.getConnection(); }catch(Exception e){ e.printStackTrace(); } }}Trigger the “Connection Leak” problemIn the “Server” tab of Daytrader home page, select the “Jdbc Connection Leak”checkbox and click the “Update Configuration” button. It will leak the connection ofTradeDataSource at the specified frequency, and you can also modify the “LeakFrequency” accordingly.
  37. 37. Figure40: Error reproduction optionsMonitoring the “Connection Leak” problemWhen a connection leak problem occurs, you will see the red data and warning marksin the dashboard. Figure41: Monitoring the dashboardDouble click the server panel to switch to the monitoring page, and you can seeseveral connection pool and servlet alerts in the alert section. Figure42: Alert PanelGo to the detailed data page to check the alert content.
  38. 38. Figure43: Detailed alert contentGo to connection pool detailed page to check the c Figure44: Detailed connection pool dataSome other problems will be triggered after a while.
  39. 39. Figure45: Monitoring the dashboardDetermine the “Connection Leak” problemIn order to get the jdbc connection allocation stack trace, we need to enable the“ConnLeakLogic” trace. In the “topology” view, right click the “sick” server andselect the “Enable Trace…” from the context menu.In the popup dialog box, select the “ConnLeakLogic” trace and click “Ok”, button toenable the trace.
  40. 40. Figure46: Enable the “ConnLeakLogic” traceAfter several minutes, right click the “sick” server again and select the “Showconnection pool contents” from the context menu. Figure47: Show connection pool contentsIn the connection pool content window, select the “TradeDataSource”, you will beable to check all the connections status, including the allocation time, used time, usedthread number and allocation stack trace.Note: If you did not enable the “ConnLeakLogic” trace, the allocation stack trace willnot be able to shown here. Besides, we only be able to get the alloction stack traceinfo for connections that allocated after we enable the “ConnLeakLogic” trace, otherconnections’ allocation stack trace is not recorded.
  41. 41. Figure48: Connection pool contentsFrom the connection pool content info, we will be able to see that lots of connectionsare used for more then 200 seconds by some threads, but all of them can not be foundin the thread dump, which means that the threads are completed but the connectionsare not released, so it is a connection leak problem. Besides, nearly all of the leakedconnections were allocated by the following stack trace, so we can fix the problem inour code accordingly.
  42. 42. Scenario5: “Memory Leak” problemApplication Changes to Simulate “Memory Leak” SituationIn this scenario, some requests will put some byte[] to the static ArrayList which willnever be released. You can adjust the “Leak Size” to control the leak rate.Code changes in the TradeAppServlet//For Memory Leakpublic static ArrayList leakContainer = new ArrayList();if(TradeConfig.isMemeoryLeak()){ leakContainer.add(new byte[TradeConfig.getLeakSize()]); }Trigger the “Memory Leak” problemIn the “Server” tab of Daytrader home page, select the “Memory Leak” checkbox andclick the “Update Configuration” button. It will leak the heap memory at the specifiedrate.
  43. 43. Figure49: Error reproduction optionsMonitoring the “Memory Leak” problemWhen a connection leak problem occurs, you will see the red data and warning marksin the dashboard.
  44. 44. Figure50: Monitoring the dashboardDouble click the server panel to switch to the monitoring page, and you can seeruntime alerts in the alert section. Figure51: Alert PanelClick the alert point to check the alert content in the detailed data page. Figure52: Detailed Alert ContentsDetermine the “Memory Leak” problemIn order to determine the memory leak problem, we need to generate a heap dump. Inthe “topology” view, right click the “sick” server and select the “Generate heapdump” from the context menu. The heap dump file will be generated in the profilehome by default.
  45. 45. Figure53: Generate a heap dumpThen add the heap dump into the "Case" in the ISA v5. Figure54: Add heapdump file to you caseIn "Tools" panel, Launch "Memory Analyzer" for JVM heap dump.
  46. 46. Figure55: Analyze the heap dumpClick "Launch", then Click "Browse" to select a dump file. Submit the task. Figure56: Start to analyzeThe task is running as a background task. A pop-up window appears to indicate theoutput directory: Figure57: Confirm dialog
  47. 47. The execution history and status are shown in the bottom of "memory analyzer" page. Figure58: Status panelMemory analyzer takes several minutes to analyze the dump file. Please be patientwhen the task is running background.Check the output directory for the heapdump analysis and open the report file"heapdump.*_analyzer.html". Figure59: Check the analysis resultsSelect the "Leak suspects" reports for details. In this sample, "problem suspect 1"consumed 237.4MB memory.
  48. 48. Figure60: Leak suspectsExpand the detail analysis for problem suspect 1, you will find the memory issue isintroduced by "ArrayList" objects in TradeAppServlet, which is the code changes wedid in section Application Changes to Sils -lmulate "Memory Leak" Situation.
  49. 49. Figure61: Detailed info of leak suspectsFor more information about Memory Analyzer, visit http://www.ibm.com/developerworks/java/jdk/tools/index.htmlOther optional functions you can go throughTuning the performance related parametersThere are 2 ways to change the settings of application server in the PTT. The first oneis via the “Tuning Parameters” view.
  50. 50. Figure62: Tune the perfomance parametersThe second way is executing tuning scripts in the scripts view. Some useful tuningscripts have been added to the toolkit and you can make some modification base onthem. Figure63: Run wsadmin scriptsAdjust rules from rule editor if needed
  51. 51. The WebSphere Application Server Performance Tuning Toolkit (PTT) allows users to define theirown rules according to the indicator of their system. For example, set a proper threshold forresponse time of each servlet. All the customized rules for detecting performance decline andlocating the bottleneck are write in the rule file.The existing rules:· If the used heap reaches the 85% of maximum heap size, it will generate a runtime alert· If the used thread pool reaches 90% of the max pool size, it will generate a threadpool alert· If the average CPU usage reaches 90%, it will generate a runtime alert· If some servlet errors occur, it will generate a servlet alert· If some JDBC connection timeout occur, it will generate a connection alert· If there are more then 1000 prepared statement discarded, it will generate a connection alert· If there are some thread waiting for connection, it will generate a connection alert· If some JCA connection error occur, it will generate a JCA alert· If there is no room for new session, it will generate a session alert· If some hung threads are declared, it will generate a threadpool alert· If some transactions rolled back, it will generate a transaction alert· If some transactions timeout, it will generate a transaction alert· If some proxy request failed, it will generate a proxy alert· If a servlet performance decline detect it will generate a servlet alter· If a jdbc performance problem detect, it will generate a connection pool alertAll these rules are common rules, and you may need to modify them according to thecharacters of your own system. For example, we can change the threshold of servletresponse time according to the performance data we collected to make theperformance problem detection rule more precise.
  52. 52. Figure64: Modify the rulesGenerate reportYou can also generate a report to get a clear view of your system. Figure65: Generate a report More Functions will be added later ……

×