Tools and Tips to Diagnose Performance Issues

13,614 views

Published on

This is the presentation I did at Jazoon 2008 conference.

Published in: Technology
1 Comment
11 Likes
Statistics
Notes
No Downloads
Views
Total views
13,614
On SlideShare
0
From Embeds
0
Number of Embeds
1,986
Actions
Shares
0
Downloads
805
Comments
1
Likes
11
Embeds 0
No embeds

No notes for slide

Tools and Tips to Diagnose Performance Issues

  1. 1. Tools and Tips to Diagnose Performance Issues Claudio Miranda Summa Technologies 3320
  2. 2. 2 Objective Learn some tips and tools to better diagnose and solve the most common performance issues of Java applications. The main focus going on to diagnose memory and CPU problems.
  3. 3. 3 Disclaimer The presented tools are based on the author's experience. These arguments can be applied to development and production systems
  4. 4. 4 Disclaimer 2 There are plenty of others free and commercial tools available, some better suited to specific tasks, other tools can be easy or complex to use, it all depends of: Time Money
  5. 5. 5 AGENDA > Where it all begins > Tools and tips to diagnose problems > Application and system performance > General tips
  6. 6. 6 AGENDA > Where it all begins > Tools and tips to diagnose problems > Application and system performance > General tips
  7. 7. 7 Where it all begins > No performance requirements at project level > Traps – “The application is too slow” – “Speed it up at all costs” How much to tune ? It is necessary to have performance numbers to work with > Typical numbers – Tx/s (Average transactions per second) – Concurrent users – Number of sessions
  8. 8. 8 Where it all begins > What is a transaction ? – Database – Messaging – Connector – Filesystem – Is “two phase commit” required ? > Call it a business transaction – Example: 2 DB inserts + 1 messaging – Identify the hot spots
  9. 9. 9 Where it all begins > Where is the problem ? – Memory – I/O – Networking – CPU >
  10. 10. 10 Where it all begins > Personal diagnose and solve steps – Analysis => What is the matter ? – Priority => Is the application down or slow ? – Diagnose => Where is the problem ? – Solution => How to solve it ?  Sledgehammer way  Best practices – Tests => Make sure it will not happen again – Documentation => To remember what happened
  11. 11. 11 AGENDA > Where it all begins > Tools and tips to diagnose problems > Application and system performance > General tips
  12. 12. 12 Tools and Tips > Memory issues – Memory Leak – OutOfMemoryError (OOME) > CPU issues – High CPU usage – Threads > Garbage Collector
  13. 13. 13 Tools and Tips Each tool and tip has different level of intrusion on the overall system. It ranges from simple monitoring (with ready to use tools) to profiling and debugging. That is one key point, to consider which approach to use, to diagnose a problem
  14. 14. 14 Tools and Tips > Monitoring – Non intrusive – No pre-configuration – Monitor production systems – Easy to use > Profiling and Debugging – Easily shows where is the problem – Nice graphics – Needs to restart system – Slowdown system runtime – Additional settings – Can distort results
  15. 15. 15 Tools and Tips Diagnose Memory Issues
  16. 16. 16 Tools and Tips - Memory > Required steps – How much physical memory is available – How many programs are using it – Who is the memory hunger ?  What are the VM parameters ? – Application did entire life cycle to prevent new classes being loaded  Initial load time
  17. 17. 17 Tools and Tips – Monitoring Memory > How much Physical memory > Who is the memory hunger > Monitoring: top – RES = Physical memory
  18. 18. 18 Tools and Tips – Monitoring Memory Java heap space > Monitoring: jmap > GC Algorithm > Heap size
  19. 19. 19 Tools and Tips – Monitoring Memory Java heap space > Monitoring: jstat > GC Activity
  20. 20. 20 Tools and Tips – Monitoring Memory Each GC releases less memory than before Memory Leak indicator
  21. 21. 21 Tools and Tips - Memory > Track down where is the problem – Heap dump (non intrusive)  Generate a dump file as large as the heap memory  Freezes application while dump is generated – Profilers (intrusive)  Need to reconfigure VM settings  Attach a client profiler to introspect
  22. 22. 22 Tools and Tips - Memory > Heap dump > jmap -dump:live,format=b,file=dump23855_appserv1 23855 > -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=heapdump.hprof
  23. 23. 23 Tools and Tips - Memory > Heap dump and profilers > Dump Analyzers and Profilers – IBM Heap Dump Analyzer – Eclipse Memory Analyzer – NetBeans Profiler
  24. 24. 24 Tools and Tips - Memory > IBM Heap Analyzer
  25. 25. 25 Tools and Tips - Memory > IBM Heap Analyzer
  26. 26. 26 Tools and Tips - Memory > Eclipse Memory Analyzer
  27. 27. 27 Tools and Tips - Memory > NetBeans Profiler 1 2
  28. 28. 28 Tools and Tips - Memory > OutOfMemoryError – Look at log files – Search for hprof, hs_err or core files – Monitor memory usage  prstat, top, ps – Monitor GC activity  jstat, -verbose:gc, visualgc, jconsole,
  29. 29. 29 Tools and Tips - Memory > OutOfMemoryError types – Java heap space – PermGen space – Out of swap space – unable to create new native thread – Requested array size exceeds VM limit – GC overhead limit exceeded Exception in thread quot;mainquot; java.lang.OutOfMemoryError: Java heap space at ConsumeHeap$BigObject.<init>(ConsumeHeap.java:22) at ConsumeHeap.main(ConsumeHeap.java:41)
  30. 30. 30 Tools and Tips - Memory Java heap space > Causes – Low RAM memory – Small heap – Memory leak > Fix – More physical memory – Make heap larger – Fix memory leaks
  31. 31. 31 Tools and Tips - Memory PermGen space > Causes – Low memory for permanent generation area – Classes on permanent generation are prevented to be collected  Classloader leak > Fix – Make permanent generation area, larger – Disable hot deployment or class reloading – for classloader leak
  32. 32. 32 Tools and Tips - Memory Out of swap space > Cause – Often is caused by JNI code – No memory available to operating system > Fix – Reserve more memory to operating system – Lower the heap size
  33. 33. 33 Tools and Tips - Memory Unable to create new native thread > Cause – Big stack size (-Xss) – Small memory available – Poor thread management > Fix – Make stack size smaller – Reserve more memory to operating system – Lower heap size – Better thread management  Java 5 thread pool
  34. 34. 34 Tools and Tips - Memory Requested array size exceeds VM limit > Cause – Attempt to allocate an array that is larger than the heap size > Fix – Lower the array size – Make heap larger
  35. 35. 35 Tools and Tips - Memory GC overhead limit exceeded > Cause – Too much time spent on GC (98%) and little heap (2%) to be free > Fix – Similar to “java heap space” – Tune GC behavior
  36. 36. 36 Tools and Tips - Memory > VM Crashes – Search for hprof, hs_err or core files > Enable post morten diagnostics gcore -o <file> PID -XX:+HeapDumpOnOutOfMemoryError -XX:OnError > Generate thread and heap dumps to analyze it later
  37. 37. 37 Tools and Tips - CPU Diagnosing CPU Issues
  38. 38. 38 Tools and Tips - CPU > OS diagnostic tool – prstat, ps, top > Thread behavior – Thread dump, thread analyzer > GC Activity (again) – Monitor heap > Profiling
  39. 39. 39 Tools and Tips - CPU Everybody waiting this guy ! > Thread dump and analyzer
  40. 40. 40 Tools and Tips - CPU > Thread dump and analyzer
  41. 41. 41 Tools and Tips - CPU > Thread dump and analyzer
  42. 42. 42 Tools and Tips - CPU > Thread dump and analyzer
  43. 43. 43 Tools and Tips - CPU > Profiler > CPU intensive task
  44. 44. 44 Tools and Tips - CPU > Profiler > CPU intensive task, the caller class
  45. 45. 45 Tools and Tips - CPU GC Activity Too much GC in short period of time, degrades performance > Fix – Adjust GC – Make heap larger
  46. 46. 46 Tools and Tips – More tools More Tools to help diagnose issues... ...or to mess around your toolbox
  47. 47. 47 Tools and Tips – More tools > Dtrace (Solaris, FreeBSD, Mac OSX) – Examine program behavior by reading points of instrumentation on java and solaris. > Sun Studio Tools (Solaris and Linux) – Graphical tools to help diagnose and debug memory and cpu issues > Profilers: Jprobe, Optimize It, YourKit, (commercial) > BEA JRockit Memory Leak Detector > JBoss Profiler (collect first, analyze later) > JXInsight (commercial - free dev. edition) – Monitor, diagnose and analysis of java systems >
  48. 48. 48 Tools and Tips – More tools > CPU and Memory statistics – dstat, prstat, vmstat, mpstat > Networking – Wireshark (AKA ethereal), tcpdump, snoop, dig, netstat, > System call tracing – truss, strace, dtrace > Threading – pstack > Process – lsof, ps,
  49. 49. 49 AGENDA > Where it all begins > Tools and tips to diagnose problems > Application and system performance > General tips
  50. 50. 50 Application and System Performance Some general tips to improve overall performance
  51. 51. 51 Application and System Performance > Application Servers – Properly configure pool resources  Threads pools are used everywhere  Connection pools for Http, JDBC, JMS, JCA, JMX, EJBs Monitor TCP connections and blocking connections to have a  general view to tune those parameters Many threads waiting on a single monitor, adjust the pool  – Http services  Enable resource cache  Configure timeout setting for http, database, session  Enable keep-alive  Put the data where it belongs Static data on webservers  Dynamic data on appserver  – Disable DNS lookup
  52. 52. 52 Application and System Performance > Application Servers – Clustering  When session (HttpSession and SFSB) must synchronize themselves ?  Application must be designed to work under cluster  Make sure there are no-firewall between nodes  Hardware load balancers are faster  Large session size, can bring down the cluster  Memory replication is better – Classloader and Dynamic loading  Disable redeploy, autodeploy, auto discovery and JSP compilation
  53. 53. 53 Application and System Performance > Java Concurrent Programming – Use thread pools – Lower VM stack size (-Xss) – Use util.concurrent classes  Collections  Atomic operations  Framework for multithread system – Synchronize the smallest possible code – Parallelize tasks on SMP systems  Be careful, context switching can be expensive
  54. 54. 54 Application and System Performance > Java Runtime – Disable security manager at trusted sites – Disable class verifier (-Xnoverify) – Garbage Collector  Use Parallel GC when possible  Disable verbose:gc, prefer jstat 32 – 32 bit systems have address space restrictions of 2 bytes  Cannot configure -Xmx4g  If possible don't enable 64 bits (-d64)
  55. 55. 55 Application and System Performance > Database – Have a database expert – For report systems  Configure a specific connection pool  Set a less restrictive transaction isolation level – Data cache  A cleaning policy is highly recommended – Don't use XA protocol, if possible – Use stored procedures for:  Data transformation between datasets  For large datasets – Pure JDBC for reporting or complex queries – Create indexes – Use optimistic concurrency scheme
  56. 56. 56 Application and System Performance > I/O System – Get rid of all System.out – Use “if (log.isDebugEnabled())” before logging statements, to prevent string concatenation – Prefer java.io.Externalizable over standard serialization – Use buffers on I/O operations – On NIO systems  Be careful of allocation outside heap, use -XX:MaxDirectMemorySize  Decide which type of allocation to use  Consider FileChannel for copy or read operations
  57. 57. 57 Application and System Performance > General Java Programming – Use Weak and Soft references – Bad hashCode and equals implementation can make your cache crazy > XML – Prefer a binary formato for communication protocol > Log management – Log levels for production and development systems – No debug messages for production systems, please – Don't log method names or package abbreviation
  58. 58. 58 AGENDA > Where it all begins > Tools and tips to diagnose problems > Application and system performance > General tips
  59. 59. 59 General Tips Why is the top management upset at stacktraces on the server ?
  60. 60. 60 General Tips > Don't panic yourself into the crazy bandwagon > Try to have a good view of the overall computing system – Infrastructure, operating system, daemons, firewalls, application servers, configurations, what was working before > Be careful to trust the information sources > Don't prematurely tell the customer about the solution > Critical understanding of the affected services and importance to the customer > Get to know teams and respective leaders/managers > Confirm reports told by other people > Verify system settings
  61. 61. 61 General Tips > Understand how to use the tools and its output > Don't depend on graphical consoles > Have access to the application's source code > Have a good decompiler (if the previous one fails) > When notifying the customer about the issue, bring him the fix – And have a happy customer
  62. 62. 62 Questions ?
  63. 63. Claudio Miranda http://weblogs.java.net/blog/claudio claudio@claudius.com.br Summa Technologies http://www.summa-tech.com

×