Analysis bottleneck in J2EE application
Upcoming SlideShare
Loading in...5
×
 

Analysis bottleneck in J2EE application

on

  • 3,787 views

How to find bottleneck and tuning in j2ee application by using java thread dump, and guide for common mistake of j2ee developer

How to find bottleneck and tuning in j2ee application by using java thread dump, and guide for common mistake of j2ee developer

Statistics

Views

Total Views
3,787
Views on SlideShare
3,771
Embed Views
16

Actions

Likes
5
Downloads
264
Comments
0

2 Embeds 16

http://www.slideshare.net 11
http://www.linkedin.com 5

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Analysis bottleneck in J2EE application Analysis bottleneck in J2EE application Presentation Transcript

  • BEA Systems Korea Byungwook Cho (bcho@bea.com) Analysis WebLogic Hang up & slow down
  • Agenda What is slowdown & hang up? 1 Slowdown in WAS 2 Slowdown in other resources 3 Common mistake in J2ee developer 4
  • What is the slowdown & hang up?
    • Slowdown
      • System runs very slowly like a stop
      • Sometimes, system gets a lot of resources (CPU,Memory)
    • Hang up
      • System stops
      • System has no response
  • What is the slowdown & hang up
    • To solve this problem
    Find the reason Solve it!! With experience,tool and test Find the problem Development env : Stress Test (Load Runner,MS Stress -free) Production : Itself is a problem
  • What is the slowdown & hang up User AP WAS JVM DBMS Web Server Network Client < Basic J2ee architecture > Legacy
  • Slow down in WAS & User AP
  • Slow down in WAS & User AP User AP WAS JVM DBMS Web Server Network Client Legacy
  • Slow down in WAS & User AP
    • Thread Pooling
    Thread Pool Idle Thread (Waiting) request Thread Pool Working Thread (Handling request) request response Thread Pool Working Thread
  • Slow down in WAS & User AP
    • Basic structure of WAS
    WebServer OR Browser OR Client Dispatcher Request Queue Thread Pool Connection Pool Thread Queue JMS Thread Queue APP1 Thread Queue APP2 Thread Queue Other Resources Connector Working Thread Idle Thread
  • Slow down in WAS & User AP
    • Java Thread State
    Runnable new start Blocked Sleep/done sleep Wait/notify Suspend/Resume Block on IO/IO complete dead stop << Thread State Diagram >> - Runnable : running -  Suspended,Locked : waiting on monitor entry / waiting on condition /MW - Wait : Object.Wait()
  • Slow down in WAS & User AP
    • Thread dump
      • Snapshot of java ap (X-ray)
      • We can recognize thread running tread state by “continuous thread dump”
  • Slow down in WAS & User AP
    • Getting Thread Dump
      • Unix : kill –3 pid , Windows : Ctrl + Break
      • Get thread dump about 3~5 times between 3~5secs
      • ※ Linux : pstree -pan
    Threads Time Working thread Thread dump
  • Slow down in WAS & User AP
    • Thread dump
      • It displays all of thread state by “THREAD STACK”
      &quot;ExecuteThread: '42' for queue: 'default'&quot; daemon prio=5 tid=0x3504b0 nid=0x34 runnable [0x9607e000..0x9607fc68] Thread name Thread id (signature) Thread State Program stack of this thread Sun JVM at java.net.SocketInputStream.read(SocketInputStream.java:85) at oracle.net.ns.Packet.receive(Unknown Source) at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.java:730) at oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.java:702) at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:373) at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1427) at oracle.jdbc.ttc7.TTC7Protocol.fetch(TTC7Protocol.java:911) at oracle.jdbc.driver.OracleStatement.doExecuteQuery(OracleStatement.java:1948) at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:2137) at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:404) at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:344) at weblogic.jdbc.pool.PreparedStatement.executeQuery(PreparedStatement.java:51) at weblogic.jdbc.rmi.internal.PreparedStatementImpl.executeQuery(PreparedStatementImpl.java:56)            
  • Slow down in WAS & User AP
    • How to analysis thread dump
    MYTHREAD_RUN(){   call methodA(); }   methodA(){ //doSomething();   call methodB(); }   methodB(){ //call methodC();   } methodC(){ // doSomething   for(;;){// infinite loop }   } “ MYTHREAD” runnable   at …MYTHREAD_RUN() :   “ MYTHREAD” runnable   at …methodA() at …MYTHREAD_RUN() :   “ MYTHREAD” runnable   at …methodB() at …methodA() at …MYTHREAD_RUN() :   “ MYTHREAD” runnable   at …methodC() at …methodB() at …methodA() at …MYTHREAD_RUN() :   “ MYTHREAD” runnable at …methodC() at …methodB() at …methodA() at …MYTHREAD_RUN() : 계속 이모양이 반복됨 Source code Thread dumps
  • Slow down in WAS & User AP
    • CASE 1. Lock contention
      • In the java application java thread wait for lock in synchronized method
      • Occurred in multi threaded program
      • Reduce frequency of synchronized method
      • Synchronized block must be implemented to be end as soon as possible ( ※ IO? )
    public void synchronized methodA(Object param) { //do something while(1){} } < sample code >
  • Slow down in WAS & User AP
    • CASE 1. Lock contention
    Thread1 Thread3 Thread2 Thread4 Sychronized MethodA Threads Time Thread1 – That has a lock Thread 2.3.4 – waiting for lock
  • Slow down in WAS & User AP
    • CASE 1. Lock contention : Thread dump example
    &quot;ExecuteThread: '12' for queue: 'weblogic.kernel.Default'&quot; daemon prio=10 tid=0x0055ae20 nid=23 lwp_id=3722788 waiting for monitor entry [0x2fb6e000..0x2fb6d530] : at java.lang.ClassLoader.loadClass(ClassLoader.java:255) : at org.apache.xerces.jaxp.SAXParserFactoryImpl.newSAXParser(Unknown Source) at org.apache.axis.utils.XMLUtils.getSAXParser(XMLUtils.java:252) - locked < 0x329fcf50 > (a java.lang.Class)   &quot;ExecuteThread: '13' for queue: 'weblogic.kernel.Default'&quot; daemon prio=10 tid=0x0055bde0 nid=24 lwp_id=3722789 waiting for monitor entry [0x2faec000..0x2faec530] at org.apache.axis.utils.XMLUtils.getSAXParser(XMLUtils.java:247) - waiting to lock < 0x329fcf50 > (a java.lang.Class) : &quot;ExecuteThread: '15' for queue: 'weblogic.kernel.Default'&quot; daemon prio=10 tid=0x0061dc20 nid=26 lwp_id=3722791 waiting for monitor entry [0x2f9ea000..0x2f9ea530] at org.apache.axis.utils.XMLUtils.releaseSAXParser(XMLUtils.java:283) - waiting to lock < 0x329fcf50 > (a java.lang.Class) at
  • Slow down in WAS & User AP
    • CASE 2. Dead lock
      • CWC “Circular waiting condition”
      • Commonly it makes WAS hang up
      • Remove CWC by changing lock direction
      • Some kind of JVM detects dead lock automatically
    sychronized methodA(){ call methodB(); } sychronized methodB(){ call methodC(); } sychronized methodC(){ call methodA(); } < sample code >
  • Slow down in WAS & User AP
    • CASE 2. Dead lock
    Thread1 Sychronized MethodA Thread2 Sychronized MethodB Thread3 Sychronized MethodC Threads Time Thread 1 Thread 2 Thread 3 Circular waiting condition
  • Slow down in WAS & User AP
    • CASE 2. Dead lock
    FOUND A JAVA LEVEL DEADLOCK: ---------------------------- &quot;ExecuteThread: '12' for queue: 'default'&quot;:   waiting to lock monitor 0xf96e0 (object 0xbd2e1a20, a weblogic.servlet.jsp.JspStub ),   which is locked by &quot;ExecuteThread: '5' for queue: 'default'&quot; &quot;ExecuteThread: '5' for queue: 'default'&quot;:   waiting to lock monitor 0xf8c60 (object 0xbd9dc460, a weblogic.servlet.internal.WebAppServletContext ),   which is locked by &quot;ExecuteThread: '12' for queue: 'default'&quot; JAVA STACK INFORMATION FOR THREADS LISTED ABOVE: ------------------------------------------------ Java Stack for &quot;ExecuteThread: '12' for queue: 'default'&quot;: ==========   at weblogic.servlet.internal.ServletStubImpl.destroyServlet(ServletStubImpl.java:434)  - waiting to lock <bd2e1a20> (a weblogic.servlet.jsp.JspStub)  at weblogic.servlet.internal.WebAppServletContext.removeServletStub(WebAppServletContext.java:2377)  at weblogic.servlet.jsp.JspStub.prepareServlet(JspStub.java:207) :  Java Stack for &quot;ExecuteThread: '5' for queue: 'default'&quot;: ==========  at weblogic.servlet.internal.WebAppServletContext.removeServletStub(WebAppServletContext.java:2370)  at weblogic.servlet.jsp.JspStub.prepareServlet(JspStub.java:207)   at weblogic.servlet.jsp.JspStub.prepareServlet(JspStub.java:154)  - locked <bd2e1a20> (a weblogic.servlet.jsp.JspStub)  at weblogic.servlet.internal.ServletStubImpl.getServlet(ServletStubImpl.java:368)  :   Found 1 deadlock.======================================================== Deadlock auto detect in Sun JVM
  • Slow down in WAS & User AP
    • CASE 2. Dead lock
    &quot;ExecuteThread-6&quot; (TID:0x30098180, sys_thread_t:0x39658e50, state:MW , native ID:0xf10) prio=5 at oracle.jdbc.driver.OracleStatement.close (OracleStatement.java(Compiled Code)) at weblogic.jdbc.common.internal.ConnectionEnv.cleanup(ConnectionEnv.java(Compiled : &quot;ExecuteThread-8&quot; (TID:0x30098090, sys_thread_t:0x396eb890, state:MW , native ID:0x1112) prio=5 at oracle.jdbc.driver.OracleConnection .commit(OracleConnection.java(Compiled Code)) at weblogic.jdbcbase.pool.Connection.commit(Connection.java(Compiled Code)) : sys_mon_t:0x39d75b38 infl_mon_t: 0x39d6e288: oracle.jdbc.driver.OracleConnection @310BC380/310BC388: owner &quot; ExecuteThread-8 &quot; (0x396eb890) 1 entry  1) Waiting to enter: &quot;ExecuteThread-10&quot; (0x3977e2d0) &quot; ExecuteThread-6 &quot; (0x39658e50) sys_mon_t:0x39d75bb8 infl_mon_t: 0x39d6e2a8: oracle.jdbc.driver.OracleStatement@33AA1BD0/33AA1BD8 : owner &quot; ExecuteThread-6 &quot; (0x39658e50) 1 entry  2) Waiting to enter: &quot; ExecuteThread-8 &quot; (0x396eb890 Deadlock trace in AIX JVM
  • Slow down in WAS & User AP
    • CASE 3. Wait for IO Response
      • Situation in waiting for IO response (DB,Network)
      • Commonly occurred caused by DB locking or slow response
    Threads Time Wait for IO response
  • Slow down in WAS & User AP
    • CASE 3. Wait for IO Response
    &quot;ExecuteThread: '42' for queue: 'default'&quot; daemon prio=5 tid=0x3504b0 nid=0x34 runnable [0x9607e000..0x9607fc68] at java.net.SocketInputStream.socketRead(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:85) at oracle.net.ns.Packet.receive(Unknown Source) at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.java:730) at oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.java:702) at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:373) at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1427) at oracle.jdbc.ttc7.TTC7Protocol.fetch(TTC7Protocol.java:911) at oracle.jdbc.driver.OracleStatement.doExecuteQuery(OracleStatement.java:1948) at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:2137) at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:404) at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:344) at weblogic.jdbc.pool.PreparedStatement.executeQuery(PreparedStatement.java:51) at weblogic.jdbc.rmi.internal.PreparedStatementImpl.executeQuery(PreparedStatementImpl.java:56) at weblogic.jdbc.rmi.SerialPreparedStatement.executeQuery(SerialPreparedStatement.java:42) at com.XXXX 생략 at ……..
  • Slow down in WAS & User AP
    • CASE 4. High CPU usage
      • When monitoring CPU usage by top,glance. WAS process consumes a lot of CPU - almost 90~100%
      • Find it using tools below
        • Sun : prstat,pstack,thread dump
        • AIX : ps –mp,dbx,thread dump
        • HP : glance,thread dump
        • See a document “How to find bottleneck in J2EE application “ in http://www.j2eestudy.co.kr
  • Slow down in WAS & User AP
    • CASE 4. High CPU usage
      • Example in HP UX
    HP glance
    • Start “glance”
    • Press “G” and enter the WAS PID
    • Find the TID that consumes a lot of CPU resource
    • Get a thread dump
    • Find the thread in Thread dump that has a matched LWID with this TID
    • And find the reason and fix it
  • Slow down in WAS & User AP
    • How to prevent slow down & hang up
      • Use clustering (use more than 2 WAS instance)
      • Make “Dedicate thread group” for each work
      • Plz make “Good Application that has no risk!!”
  • Slow down in WAS & User AP
    • Summarize
      • Thread dump is snapshot of Java Application.
      • If u meet a slowdown situation, Get a thread dump.
      • Analysis thread dump.
      • And fix it..
  • Slow down in other resources
  • Slow down in other resources User AP WAS JVM DBMS Web Server Network Client Legacy
  • Slow down in other resources
    • Slow down in JVM
      • JVM memory layout
      • Minor GC,Full GC
  • Slow down in other resources
    • Slow down in JVM
      • Full GC takes a lot of time and it freeze the WAS.
      • In the high load, after full GC finished, request can be gathered suddenly.
      • Reduce un-needed memory usage
      • JVM Memory tuning is needed
  • Slow down in other resources
    • Slow down in WebServer
      • Webserver doesn’t make a slow down almost time
      • Use Keep-Alive between WebServer and Client
      • ※ Apache Keep Alive http:// httpd .apache.org/docs-2.0/mod/core.html# keepalive
  • Slow down in other resources
    • Slow down in network
      • Get a enough bandwidth
      • Tuning TCP/IP Kernel parameter in OS  some times it brings slowdown
        • Tuning buffer size
        • Connection abort secs etc.
        • This parameter is provided from WAS vendor
        • ※ WebLogic TCP/IP Kernel Parameter
        • http://e-docs. bea .com/ wls /docs81/ plugins / plugin _ params .html#1143055
  • Slow down in other resources
    • Slow down in DBMS
      • Tune DBMS (It’s not a issue of this session)
      • We can profiling execute time of SQL
      • ※ h ttp://eclipse.new21.org/phpBB2/viewtopic.php?printertopic=1&t=380&start=0&postdays=0&postorder=asc&vote=viewresult
      • ※ http://www.j2eestudy.co.kr/qna/bbs_read.jsp?table=j2ee&db=qna0104&id=5&searchBy=subject&searchKey=sql&block=0&page=0
  • Common mistake in J2ee developer
  • Common mistake in J2ee developer
    • Java Programming
    • Servlet/JSP Programming
    • JDBC Programming
    • EJB Programming
  • Common mistake in J2ee developer
    • Common mistake in Java Programming
      • Do not use String class in complex string operation
      • Close socket and file after use it
      • Use synchronized block appropriately
  • Common mistake in J2ee developer
    • Common mistake in Servlet/JSP programming
      • JSP buffer size
        • <% page buffer=“128kb” %>
      • Member variable
        • <%! Parameter param; %>
        • Use synchronized mehtod
      • Out of memory in file uploading
  • Common mistake in J2ee developer
    • Common mistake in JDBC programming
      • Close stmt, pstmt when using Connection Pooling
      • Out of memory in Bigsize query result
        • Do not save the result in memory (Vector etc..)
      • Connection Leak
    conn = getConnection(); try{ do something }catch(Exception e){ dosomething(); } }finally{ if(stmt!=null) stmt.close(); if(conn!=null) conn.close(); }
  • Common mistake in J2ee developer
    • Common mistake in EJB programming
      • Study and use EJB in appropriate place
      • Remove EJB after use it (SFSB)
      • Reduce JNDI look up
        • Caching Home Interface
      • Do not use!! Rollback & Commit in CMT.
      • Do not!! use hot deploy in production mode
  • Thanx [email_address]