Find bottleneck and tuning in Java Application

BEA Systems Korea Byungwook Cho (bcho@bea.com) Analysis WebLogic Hang up & slow down

Agenda What is slowdown & hang up? 1 Slowdown in WAS 2 Slowdown in other resources 3 Common mistake in J2ee developer 4

What is the slowdown & hang up?

Slowdown

System runs very slowly like a stop

Sometimes, system gets a lot of resources (CPU,Memory)

Hang up

System stops

System has no response

What is the slowdown & hang up

To solve this problem

Find the reason Solve it!! With experience,tool and test Find the problem Development env : Stress Test (Load Runner,MS Stress -free) Production : Itself is a problem

What is the slowdown & hang up User AP WAS JVM DBMS Web Server Network Client < Basic J2ee architecture > Legacy

Slow down in WAS & User AP

Slow down in WAS & User AP User AP WAS JVM DBMS Web Server Network Client Legacy

Thread Pooling

Thread Pool Idle Thread (Waiting) request Thread Pool Working Thread (Handling request) request response Thread Pool Working Thread

Basic structure of WAS

WebServer OR Browser OR Client Dispatcher Request Queue Thread Pool Connection Pool Thread Queue JMS Thread Queue APP1 Thread Queue APP2 Thread Queue Other Resources Connector Working Thread Idle Thread

Java Thread State

Runnable new start Blocked Sleep/done sleep Wait/notify Suspend/Resume Block on IO/IO complete dead stop << Thread State Diagram >> - Runnable : running - Suspended,Locked : waiting on monitor entry / waiting on condition /MW - Wait : Object.Wait()

Thread dump

Snapshot of java ap (X-ray)

We can recognize thread running tread state by “continuous thread dump”

Getting Thread Dump

Unix : kill –3 pid , Windows : Ctrl + Break

Get thread dump about 3~5 times between 3~5secs

※ Linux : pstree -pan

Threads Time Working thread Thread dump

Thread dump

It displays all of thread state by “THREAD STACK”

"ExecuteThread: '42' for queue: 'default'" daemon prio=5 tid=0x3504b0 nid=0x34 runnable [0x9607e000..0x9607fc68] Thread name Thread id (signature) Thread State Program stack of this thread Sun JVM at java.net.SocketInputStream.read(SocketInputStream.java:85) at oracle.net.ns.Packet.receive(Unknown Source) at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.java:730) at oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.java:702) at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:373) at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1427) at oracle.jdbc.ttc7.TTC7Protocol.fetch(TTC7Protocol.java:911) at oracle.jdbc.driver.OracleStatement.doExecuteQuery(OracleStatement.java:1948) at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:2137) at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:404) at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:344) at weblogic.jdbc.pool.PreparedStatement.executeQuery(PreparedStatement.java:51) at weblogic.jdbc.rmi.internal.PreparedStatementImpl.executeQuery(PreparedStatementImpl.java:56)

How to analysis thread dump

MYTHREAD_RUN(){   call methodA(); } methodA(){ //doSomething();   call methodB(); } methodB(){ //call methodC();   } methodC(){ // doSomething   for(;;){// infinite loop }   } “ MYTHREAD” runnable   at …MYTHREAD_RUN() : “ MYTHREAD” runnable   at …methodA() at …MYTHREAD_RUN() : “ MYTHREAD” runnable   at …methodB() at …methodA() at …MYTHREAD_RUN() : “ MYTHREAD” runnable   at …methodC() at …methodB() at …methodA() at …MYTHREAD_RUN() : “ MYTHREAD” runnable at …methodC() at …methodB() at …methodA() at …MYTHREAD_RUN() : 계속 이모양이 반복됨 Source code Thread dumps

CASE 1. Lock contention

In the java application java thread wait for lock in synchronized method

Occurred in multi threaded program

Reduce frequency of synchronized method

Synchronized block must be implemented to be end as soon as possible ( ※ IO? )

public void synchronized methodA(Object param) { //do something while(1){} } < sample code >

CASE 1. Lock contention

Thread1 Thread3 Thread2 Thread4 Sychronized MethodA Threads Time Thread1 – That has a lock Thread 2.3.4 – waiting for lock

CASE 1. Lock contention : Thread dump example

"ExecuteThread: '12' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=0x0055ae20 nid=23 lwp_id=3722788 waiting for monitor entry [0x2fb6e000..0x2fb6d530] : at java.lang.ClassLoader.loadClass(ClassLoader.java:255) : at org.apache.xerces.jaxp.SAXParserFactoryImpl.newSAXParser(Unknown Source) at org.apache.axis.utils.XMLUtils.getSAXParser(XMLUtils.java:252) - locked < 0x329fcf50 > (a java.lang.Class) "ExecuteThread: '13' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=0x0055bde0 nid=24 lwp_id=3722789 waiting for monitor entry [0x2faec000..0x2faec530] at org.apache.axis.utils.XMLUtils.getSAXParser(XMLUtils.java:247) - waiting to lock < 0x329fcf50 > (a java.lang.Class) : "ExecuteThread: '15' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=0x0061dc20 nid=26 lwp_id=3722791 waiting for monitor entry [0x2f9ea000..0x2f9ea530] at org.apache.axis.utils.XMLUtils.releaseSAXParser(XMLUtils.java:283) - waiting to lock < 0x329fcf50 > (a java.lang.Class) at

CASE 2. Dead lock

CWC “Circular waiting condition”

Commonly it makes WAS hang up

Remove CWC by changing lock direction

Some kind of JVM detects dead lock automatically

sychronized methodA(){ call methodB(); } sychronized methodB(){ call methodC(); } sychronized methodC(){ call methodA(); } < sample code >

CASE 2. Dead lock

Thread1 Sychronized MethodA Thread2 Sychronized MethodB Thread3 Sychronized MethodC Threads Time Thread 1 Thread 2 Thread 3 Circular waiting condition

CASE 2. Dead lock

FOUND A JAVA LEVEL DEADLOCK: ---------------------------- "ExecuteThread: '12' for queue: 'default'": waiting to lock monitor 0xf96e0 (object 0xbd2e1a20, a weblogic.servlet.jsp.JspStub ), which is locked by "ExecuteThread: '5' for queue: 'default'" "ExecuteThread: '5' for queue: 'default'": waiting to lock monitor 0xf8c60 (object 0xbd9dc460, a weblogic.servlet.internal.WebAppServletContext ), which is locked by "ExecuteThread: '12' for queue: 'default'" JAVA STACK INFORMATION FOR THREADS LISTED ABOVE: ------------------------------------------------ Java Stack for "ExecuteThread: '12' for queue: 'default'": ========== at weblogic.servlet.internal.ServletStubImpl.destroyServlet(ServletStubImpl.java:434) - waiting to lock <bd2e1a20> (a weblogic.servlet.jsp.JspStub) at weblogic.servlet.internal.WebAppServletContext.removeServletStub(WebAppServletContext.java:2377) at weblogic.servlet.jsp.JspStub.prepareServlet(JspStub.java:207) : Java Stack for "ExecuteThread: '5' for queue: 'default'": ========== at weblogic.servlet.internal.WebAppServletContext.removeServletStub(WebAppServletContext.java:2370) at weblogic.servlet.jsp.JspStub.prepareServlet(JspStub.java:207) at weblogic.servlet.jsp.JspStub.prepareServlet(JspStub.java:154) - locked <bd2e1a20> (a weblogic.servlet.jsp.JspStub) at weblogic.servlet.internal.ServletStubImpl.getServlet(ServletStubImpl.java:368) : Found 1 deadlock.======================================================== Deadlock auto detect in Sun JVM

CASE 2. Dead lock

"ExecuteThread-6" (TID:0x30098180, sys_thread_t:0x39658e50, state:MW , native ID:0xf10) prio=5 at oracle.jdbc.driver.OracleStatement.close (OracleStatement.java(Compiled Code)) at weblogic.jdbc.common.internal.ConnectionEnv.cleanup(ConnectionEnv.java(Compiled : "ExecuteThread-8" (TID:0x30098090, sys_thread_t:0x396eb890, state:MW , native ID:0x1112) prio=5 at oracle.jdbc.driver.OracleConnection .commit(OracleConnection.java(Compiled Code)) at weblogic.jdbcbase.pool.Connection.commit(Connection.java(Compiled Code)) : sys_mon_t:0x39d75b38 infl_mon_t: 0x39d6e288: oracle.jdbc.driver.OracleConnection @310BC380/310BC388: owner " ExecuteThread-8 " (0x396eb890) 1 entry  1) Waiting to enter: "ExecuteThread-10" (0x3977e2d0) " ExecuteThread-6 " (0x39658e50) sys_mon_t:0x39d75bb8 infl_mon_t: 0x39d6e2a8: oracle.jdbc.driver.OracleStatement@33AA1BD0/33AA1BD8 : owner " ExecuteThread-6 " (0x39658e50) 1 entry  2) Waiting to enter: " ExecuteThread-8 " (0x396eb890 Deadlock trace in AIX JVM

CASE 3. Wait for IO Response

Situation in waiting for IO response (DB,Network)

Commonly occurred caused by DB locking or slow response

Threads Time Wait for IO response

CASE 3. Wait for IO Response

"ExecuteThread: '42' for queue: 'default'" daemon prio=5 tid=0x3504b0 nid=0x34 runnable [0x9607e000..0x9607fc68] at java.net.SocketInputStream.socketRead(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:85) at oracle.net.ns.Packet.receive(Unknown Source) at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.java:730) at oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.java:702) at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:373) at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1427) at oracle.jdbc.ttc7.TTC7Protocol.fetch(TTC7Protocol.java:911) at oracle.jdbc.driver.OracleStatement.doExecuteQuery(OracleStatement.java:1948) at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:2137) at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:404) at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:344) at weblogic.jdbc.pool.PreparedStatement.executeQuery(PreparedStatement.java:51) at weblogic.jdbc.rmi.internal.PreparedStatementImpl.executeQuery(PreparedStatementImpl.java:56) at weblogic.jdbc.rmi.SerialPreparedStatement.executeQuery(SerialPreparedStatement.java:42) at com.XXXX 생략 at ……..

CASE 4. High CPU usage

When monitoring CPU usage by top,glance. WAS process consumes a lot of CPU - almost 90~100%

Find it using tools below

Sun : prstat,pstack,thread dump

AIX : ps –mp,dbx,thread dump

HP : glance,thread dump

See a document “How to find bottleneck in J2EE application “ in http://www.j2eestudy.co.kr

CASE 4. High CPU usage

Example in HP UX

HP glance

Start “glance”

Press “G” and enter the WAS PID

Find the TID that consumes a lot of CPU resource

Get a thread dump

Find the thread in Thread dump that has a matched LWID with this TID

And find the reason and fix it

How to prevent slow down & hang up

Use clustering (use more than 2 WAS instance)

Make “Dedicate thread group” for each work

Plz make “Good Application that has no risk!!”

Summarize

Thread dump is snapshot of Java Application.

If u meet a slowdown situation, Get a thread dump.

Analysis thread dump.

And fix it..

Slow down in other resources

Slow down in other resources User AP WAS JVM DBMS Web Server Network Client Legacy

Slow down in JVM

JVM memory layout

Minor GC,Full GC

Slow down in JVM

Full GC takes a lot of time and it freeze the WAS.

In the high load, after full GC finished, request can be gathered suddenly.

Reduce un-needed memory usage

JVM Memory tuning is needed

Slow down in WebServer

Webserver doesn’t make a slow down almost time

Use Keep-Alive between WebServer and Client

※ Apache Keep Alive http:// httpd .apache.org/docs-2.0/mod/core.html# keepalive

Slow down in network

Get a enough bandwidth

Tuning TCP/IP Kernel parameter in OS  some times it brings slowdown

Tuning buffer size

Connection abort secs etc.

This parameter is provided from WAS vendor

※ WebLogic TCP/IP Kernel Parameter

http://e-docs. bea .com/ wls /docs81/ plugins / plugin _ params .html#1143055

Slow down in DBMS

Tune DBMS (It’s not a issue of this session)

We can profiling execute time of SQL

※ h ttp://eclipse.new21.org/phpBB2/viewtopic.php?printertopic=1&t=380&start=0&postdays=0&postorder=asc&vote=viewresult

※ http://www.j2eestudy.co.kr/qna/bbs_read.jsp?table=j2ee&db=qna0104&id=5&searchBy=subject&searchKey=sql&block=0&page=0

Common mistake in J2ee developer

Java Programming

Servlet/JSP Programming

JDBC Programming

EJB Programming

Common mistake in Java Programming

Do not use String class in complex string operation

Close socket and file after use it

Use synchronized block appropriately

Common mistake in Servlet/JSP programming

JSP buffer size

<% page buffer=“128kb” %>

Member variable

<%! Parameter param; %>

Use synchronized mehtod

Out of memory in file uploading

Common mistake in JDBC programming

Close stmt, pstmt when using Connection Pooling

Out of memory in Bigsize query result

Do not save the result in memory (Vector etc..)

Connection Leak

conn = getConnection(); try{ do something }catch(Exception e){ dosomething(); } }finally{ if(stmt!=null) stmt.close(); if(conn!=null) conn.close(); }

Common mistake in EJB programming

Study and use EJB in appropriate place

Remove EJB after use it (SFSB)

Reduce JNDI look up

Caching Home Interface

Do not use!! Rollback & Commit in CMT.

Do not!! use hot deploy in production mode

Thanx [email_address]

Find bottleneck and tuning in Java Application

0 comments

Notes on slide 1

Favorites, Groups & Events