Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Performance, profiling, and production troubleshooting tools<br />on the Java platform <br />
About CONTEXTWEB Exchange<br />Core platform written in Lightweight Java<br />10,000+ Requests per second<br /><50ms Avera...
About this talk<br />Internal Training<br />Developers – don’t always know troubleshooting<br />Ops – don’t always know Ja...
Problems We’ve Seen<br />Stability<br />CPU under load<br />System Stability<br />Scalability<br />Maximum throughput per ...
Things we look at<br />Garbage Collection<br />Thread Contention<br />Resource Usage<br />5<br />
6<br />GC<br />
JVM Garbage Collection<br />7<br />
JVM Garbage Collection<br />8<br />
JVM Garbage Collection<br />9<br />-Xloggc:/path/to/logfile (log GC activity)<br />9.302: [GC 1146880K->75819K(3002368K), ...
JVM Garbage Collection<br />10<br />
JVM Garbage Collection<br />11<br />
JVM Garbage Collection<br />12<br />
JVM Garbage Collection<br />13<br />Jmap<br />Lets you take memory snapshots and dumps against a running, or even hung JVM...
JVM (not) Garbage Collection<br />java.lang.OutOfMemoryError: Java heap space<br />java.lang.OutOfMemoryError: PermGen spa...
Garbage Collection: Tools we use<br />Verbose GC Logs – every JVM<br />Jstat – every 5 seconds<br />Jconsole - monitoring<...
Thread Contention<br />Company Confidential 16<br />Full Java Thread Dump Java HotSpot(TM) 64-Bit Server VM 19.1-b02 Sun M...
Thread Contention<br />Thread Dumps are your best friend<br />kill -3<br />Jstack (-J-d64 –l –m)<br />Look for:<br />Many ...
Thread Contention<br />Thread-dump analysis requires many different dumps<br />lsof/netstat/other system tools often provi...
Thread Contention: Tools we use<br />Thread dumps – every 15 seconds<br />Shutdown scripts <br />auto thread dump 5x<br />...
Resource Usage<br />Memory<br />Don’t forget PermGen<br />Don’t forget Stack (per thread)<br />File Descriptors<br />DB/Ne...
Resource Usage: Tools we use<br />ab (Apache Benchmark)<br />Dirt simple<br />ab -n 10000000 -k -c 200 http://localhost:80...
Conclusions<br />Many free tools give insight<br />You can never have enough historical data<br />Many default settings wi...
References<br />http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html<br />http://prefetch.net/blog/index....
Upcoming SlideShare
Loading in …5
×

NYC Java Meetup - Profiling and Performance

3,498 views

Published on

A brief overview of some of the tools that ship with the Java platform that can be used to troubleshoot performance issues, and common production/performance problems

Published in: Technology
  • Be the first to comment

NYC Java Meetup - Profiling and Performance

  1. 1. Performance, profiling, and production troubleshooting tools<br />on the Java platform <br />
  2. 2. About CONTEXTWEB Exchange<br />Core platform written in Lightweight Java<br />10,000+ Requests per second<br /><50ms Average Response Time<br />May query 40+ parties per auction<br />Multiple daily code pushes<br />Company Confidential 2<br />
  3. 3. About this talk<br />Internal Training<br />Developers – don’t always know troubleshooting<br />Ops – don’t always know Java tooling<br />Introduction to some basics and tooling distributed with the JDK<br />Company Confidential 3<br />
  4. 4. Problems We’ve Seen<br />Stability<br />CPU under load<br />System Stability<br />Scalability<br />Maximum throughput per application<br />Average Response time under load<br />Hardware utilization<br />4<br />
  5. 5. Things we look at<br />Garbage Collection<br />Thread Contention<br />Resource Usage<br />5<br />
  6. 6. 6<br />GC<br />
  7. 7. JVM Garbage Collection<br />7<br />
  8. 8. JVM Garbage Collection<br />8<br />
  9. 9. JVM Garbage Collection<br />9<br />-Xloggc:/path/to/logfile (log GC activity)<br />9.302: [GC 1146880K->75819K(3002368K), 0.0723170 secs]<br />9.819: [Full GC 152814K->75275K(3002368K), 0.2690010 secs]<br />13.829: [Full GC 368689K->79768K(3002368K), 0.3586360 secs]<br />22.635: [GC 1226648K->108287K(3002368K), 0.0190360 secs]<br />26.485: [GC 1255167K->94485K(3002368K), 0.0176960 secs]<br />27.039: [Full GC 174388K->99964K(3002368K), 0.4527120 secs]<br />34.997: [GC 1246844K->170532K(3002368K), 0.1357100 secs]<br />193.227: [GC 1317366K->244168K(3002368K), 0.1290920 secs]<br />197.768: [GC 1391023K->259567K(3002368K), 0.2124620 secs]<br />341.069: [GC 1406447K->332544K(3002368K), 0.2839650 secs]<br />346.165: [GC 1479424K->315776K(3002368K), 0.0939770 secs]<br />
  10. 10. JVM Garbage Collection<br />10<br />
  11. 11. JVM Garbage Collection<br />11<br />
  12. 12. JVM Garbage Collection<br />12<br />
  13. 13. JVM Garbage Collection<br />13<br />Jmap<br />Lets you take memory snapshots and dumps against a running, or even hung JVM<br />jmap-F -J-d64 -dump:live,format=b,file=dump.hprof JAVA_PID<br />/usr/java/default/bin/jmap<br />
  14. 14. JVM (not) Garbage Collection<br />java.lang.OutOfMemoryError: Java heap space<br />java.lang.OutOfMemoryError: PermGen space<br />java.lang.OutOfMemoryError: request <size> bytes for <reason>.<br />14<br />
  15. 15. Garbage Collection: Tools we use<br />Verbose GC Logs – every JVM<br />Jstat – every 5 seconds<br />Jconsole - monitoring<br />JvisualVM – visualization<br />Jmap – memory dumps<br />Yourkit – memory profiling<br />15<br />
  16. 16. Thread Contention<br />Company Confidential 16<br />Full Java Thread Dump Java HotSpot(TM) 64-Bit Server VM 19.1-b02 Sun Microsystems Inc.Number of threads: 545Number of daemon threads: 485Peak live thread count since the Java virtual machine started or peak was reset: 972Is support for thread contention monitoring available on this JVM? [true]Is thread contention monitoring enabled? [false]. If false, some thread synchronization statistics are not be available.Is support for CPU time measurement for any thread available on this JVM? [true]Is thread CPU time measurement enabled? [true]. If false, thread execution times are not available for any thread.--------------------------------------------------------------------------------Thread Execution Information:-----------------------Thread "http-thread-pool-8080-(862241)" thread-id: 879,948 thread-state: TIMED_WAITING Waiting on lock: java.util.concurrent.CountDownLatch$Sync@e3963a8 at: sun.misc.Unsafe.park(Native Method) at: java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at: java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011) at: java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303) at: java.util.concurrent.CountDownLatch.await(CountDownLatch.java:253) at: com.contextweb.commons.client.http.batch.httpcorenio2.BatchClient$BatchClientJob.execute(BatchClient.java:115) at: com.contextweb.commons.client.http.batch.util.AbstractBatchExecutionJob.execute(AbstractBatchExecutionJob.java:18) at: com.contextweb.adserving.rt.buyer.RTBuyerBatchExecutorImpl.executeJob(RTBuyerBatchExecutorImpl.java:48) at: com.contextweb.adserving.rt.buyer.RTBuyerBidRequestProcessor.executeJob(RTBuyerBidRequestProcessor.java:135)<br />
  17. 17. Thread Contention<br />Thread Dumps are your best friend<br />kill -3<br />Jstack (-J-d64 –l –m)<br />Look for:<br />Many objects in blocked wait<br />Threadpool usage<br />Thread/Resource pools missing items<br />Synchronization/Concurrency Issues<br />Company Confidential 17<br />
  18. 18. Thread Contention<br />Thread-dump analysis requires many different dumps<br />lsof/netstat/other system tools often provide hints on contention points<br />(historically) AppServers/libs shipped with ridiculously low defaults<br />Worker threads<br />DB connections<br />File descriptors (linux)<br />Company Confidential 18<br />
  19. 19. Thread Contention: Tools we use<br />Thread dumps – every 15 seconds<br />Shutdown scripts <br />auto thread dump 5x<br />Lsof output<br />Netstat output<br />(optional) heap dump<br />19<br />
  20. 20. Resource Usage<br />Memory<br />Don’t forget PermGen<br />Don’t forget Stack (per thread)<br />File Descriptors<br />DB/Network Connections<br />Timeouts/Expirations<br />Connection/Read default to forever<br />LDAP/DNS default to forever<br />URL – blocking DNS call<br />Company Confidential 20<br />
  21. 21. Resource Usage: Tools we use<br />ab (Apache Benchmark)<br />Dirt simple<br />ab -n 10000000 -k -c 200 http://localhost:8080/bls/check?urlreferrercheck=geI0qJVZa/Uu7;5DtxzgRilmw==<br />Siege<br />Mud simple<br />siege -c 20 http://localhost:8080/bls/check?useragenthash=adfasasdasd<br />Instrumentation<br />Home-grown – RollingCounter<br />Timers<br />Company Confidential 21<br />
  22. 22. Conclusions<br />Many free tools give insight<br />You can never have enough historical data<br />Many default settings will get you in trouble<br />Company Confidential 22<br />
  23. 23. References<br />http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html<br />http://prefetch.net/blog/index.php/2008/01/16/monitoring-garbage-collection-with-jstat/<br />http://download.oracle.com/javase/6/docs/technotes/tools/share/jstack.html<br />http://java.sun.com/developer/technicalArticles/J2SE/monitoring/<br />http://www.yourkit.com<br />

×