NYC Java Meetup - Profiling and Performance


Published on

A brief overview of some of the tools that ship with the Java platform that can be used to troubleshoot performance issues, and common production/performance problems

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

NYC Java Meetup - Profiling and Performance

  1. 1. Performance, profiling, and production troubleshooting tools<br />on the Java platform <br />
  2. 2. About CONTEXTWEB Exchange<br />Core platform written in Lightweight Java<br />10,000+ Requests per second<br /><50ms Average Response Time<br />May query 40+ parties per auction<br />Multiple daily code pushes<br />Company Confidential 2<br />
  3. 3. About this talk<br />Internal Training<br />Developers – don’t always know troubleshooting<br />Ops – don’t always know Java tooling<br />Introduction to some basics and tooling distributed with the JDK<br />Company Confidential 3<br />
  4. 4. Problems We’ve Seen<br />Stability<br />CPU under load<br />System Stability<br />Scalability<br />Maximum throughput per application<br />Average Response time under load<br />Hardware utilization<br />4<br />
  5. 5. Things we look at<br />Garbage Collection<br />Thread Contention<br />Resource Usage<br />5<br />
  6. 6. 6<br />GC<br />
  7. 7. JVM Garbage Collection<br />7<br />
  8. 8. JVM Garbage Collection<br />8<br />
  9. 9. JVM Garbage Collection<br />9<br />-Xloggc:/path/to/logfile (log GC activity)<br />9.302: [GC 1146880K->75819K(3002368K), 0.0723170 secs]<br />9.819: [Full GC 152814K->75275K(3002368K), 0.2690010 secs]<br />13.829: [Full GC 368689K->79768K(3002368K), 0.3586360 secs]<br />22.635: [GC 1226648K->108287K(3002368K), 0.0190360 secs]<br />26.485: [GC 1255167K->94485K(3002368K), 0.0176960 secs]<br />27.039: [Full GC 174388K->99964K(3002368K), 0.4527120 secs]<br />34.997: [GC 1246844K->170532K(3002368K), 0.1357100 secs]<br />193.227: [GC 1317366K->244168K(3002368K), 0.1290920 secs]<br />197.768: [GC 1391023K->259567K(3002368K), 0.2124620 secs]<br />341.069: [GC 1406447K->332544K(3002368K), 0.2839650 secs]<br />346.165: [GC 1479424K->315776K(3002368K), 0.0939770 secs]<br />
  10. 10. JVM Garbage Collection<br />10<br />
  11. 11. JVM Garbage Collection<br />11<br />
  12. 12. JVM Garbage Collection<br />12<br />
  13. 13. JVM Garbage Collection<br />13<br />Jmap<br />Lets you take memory snapshots and dumps against a running, or even hung JVM<br />jmap-F -J-d64 -dump:live,format=b,file=dump.hprof JAVA_PID<br />/usr/java/default/bin/jmap<br />
  14. 14. JVM (not) Garbage Collection<br />java.lang.OutOfMemoryError: Java heap space<br />java.lang.OutOfMemoryError: PermGen space<br />java.lang.OutOfMemoryError: request <size> bytes for <reason>.<br />14<br />
  15. 15. Garbage Collection: Tools we use<br />Verbose GC Logs – every JVM<br />Jstat – every 5 seconds<br />Jconsole - monitoring<br />JvisualVM – visualization<br />Jmap – memory dumps<br />Yourkit – memory profiling<br />15<br />
  16. 16. Thread Contention<br />Company Confidential 16<br />Full Java Thread Dump Java HotSpot(TM) 64-Bit Server VM 19.1-b02 Sun Microsystems Inc.Number of threads: 545Number of daemon threads: 485Peak live thread count since the Java virtual machine started or peak was reset: 972Is support for thread contention monitoring available on this JVM? [true]Is thread contention monitoring enabled? [false]. If false, some thread synchronization statistics are not be available.Is support for CPU time measurement for any thread available on this JVM? [true]Is thread CPU time measurement enabled? [true]. If false, thread execution times are not available for any thread.--------------------------------------------------------------------------------Thread Execution Information:-----------------------Thread "http-thread-pool-8080-(862241)" thread-id: 879,948 thread-state: TIMED_WAITING Waiting on lock: java.util.concurrent.CountDownLatch$Sync@e3963a8 at: sun.misc.Unsafe.park(Native Method) at: java.util.concurrent.locks.LockSupport.parkNanos( at: java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos( at: java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos( at: java.util.concurrent.CountDownLatch.await( at: com.contextweb.commons.client.http.batch.httpcorenio2.BatchClient$BatchClientJob.execute( at: com.contextweb.commons.client.http.batch.util.AbstractBatchExecutionJob.execute( at: com.contextweb.adserving.rt.buyer.RTBuyerBatchExecutorImpl.executeJob( at: com.contextweb.adserving.rt.buyer.RTBuyerBidRequestProcessor.executeJob(<br />
  17. 17. Thread Contention<br />Thread Dumps are your best friend<br />kill -3<br />Jstack (-J-d64 –l –m)<br />Look for:<br />Many objects in blocked wait<br />Threadpool usage<br />Thread/Resource pools missing items<br />Synchronization/Concurrency Issues<br />Company Confidential 17<br />
  18. 18. Thread Contention<br />Thread-dump analysis requires many different dumps<br />lsof/netstat/other system tools often provide hints on contention points<br />(historically) AppServers/libs shipped with ridiculously low defaults<br />Worker threads<br />DB connections<br />File descriptors (linux)<br />Company Confidential 18<br />
  19. 19. Thread Contention: Tools we use<br />Thread dumps – every 15 seconds<br />Shutdown scripts <br />auto thread dump 5x<br />Lsof output<br />Netstat output<br />(optional) heap dump<br />19<br />
  20. 20. Resource Usage<br />Memory<br />Don’t forget PermGen<br />Don’t forget Stack (per thread)<br />File Descriptors<br />DB/Network Connections<br />Timeouts/Expirations<br />Connection/Read default to forever<br />LDAP/DNS default to forever<br />URL – blocking DNS call<br />Company Confidential 20<br />
  21. 21. Resource Usage: Tools we use<br />ab (Apache Benchmark)<br />Dirt simple<br />ab -n 10000000 -k -c 200 http://localhost:8080/bls/check?urlreferrercheck=geI0qJVZa/Uu7;5DtxzgRilmw==<br />Siege<br />Mud simple<br />siege -c 20 http://localhost:8080/bls/check?useragenthash=adfasasdasd<br />Instrumentation<br />Home-grown – RollingCounter<br />Timers<br />Company Confidential 21<br />
  22. 22. Conclusions<br />Many free tools give insight<br />You can never have enough historical data<br />Many default settings will get you in trouble<br />Company Confidential 22<br />
  23. 23. References<br /><br /><br /><br /><br /><br />