Debugging Your Production JVM

  • 8,654 views
Uploaded on

This is a talk I did for JavaOne 2009. The focus of the talk was memory management and system monitoring with freely available tools that are in the jdk or open source.

This is a talk I did for JavaOne 2009. The focus of the talk was memory management and system monitoring with freely available tools that are in the jdk or open source.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • I recommend using Splunk: http://www.splunk.com/base/Community:Monitoring_JVMs
    Are you sure you want to
    Your message goes here
  • Great tip on leveraging btrace with jmx and jstat. Thanks for sharing the slides.
    Are you sure you want to
    Your message goes here
  • This is a keynote presentation... to download it and view it you would need a Mac and keynote. I will change it out for pdf.
    Are you sure you want to
    Your message goes here
  • How to download this? When i clicked getfile.. it is downloading a file with '.key' extension? What program to use to view this?
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
8,654
On Slideshare
0
From Embeds
0
Number of Embeds
3

Actions

Shares
Downloads
566
Comments
4
Likes
31

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide




















































Transcript

  • 1. Debugging Your Production JVM Machine Ken Sipe Perficient (PRFT) kensipe@gmail.com
  • 2. Abstract > Learn the tools and techniques used to monitor, trace and debugging running Java applications. 2
  • 3. Agenda > Java Memory Management > Memory Management Tools • Command-line Tools • VisualVM > Btrace > Summary > Resources 3
  • 4. Agenda > Java Memory Management > Memory Management Tools • Command-line Tools • VisualVM > Btrace > Summary > Resources 4
  • 5. Donʼt Worry… > C (malloc / free) > C++ (new / delete) > Java (new / gc) > Memory Allocation / Deallocation in Java is automatic 5
  • 6. Object Lifetimes > Most objects live very short lives • 80-98% of all newly allocated objects die • within a few million instructions • Before another megabyte is allocated > Old objects tend to live a long…. time
  • 7. Memory Spaces > New Space • Where all objects are new (sort of…) > Old Space • Where all objects are old New Space Old Space
  • 8. Memory Spaces > New Space Division • Eden • Where all objects are created • Survivor Space 0 • Provide object aging • Survivor Space 1 Eden Old SpaceSS0 SS1
  • 9. Perm Spaces > Permanent Space • class information • static information Eden Old SpaceSS0 SS1 Perm Space
  • 10. GC Responsibility > Heap Walking from GC Roots > Mark / Sweep • Garbage detection (mark) • Sort out the live ones from the dead ones • Reference counting • Garbage reclamation (sweep) • Make space available
  • 11. Minor Garbage Collection > Minor gc (scavenge) • When eden is “full” a minor gc is invoked • Sweeps through eden and the current survivor space, removing the dead and moving the living to survivor space or old • Ss0 and ss1 switch which is “current” • A new tenuring age is calculated
  • 12. Major Garbage Collection > Major gc • When old is “full” • All spaces are garbage collected including perm space • All other activities in the jvm are suspended
  • 13. Agenda > Java Memory Management > Memory Management Tools • Command-line Tools • VisualVM > Btrace > Summary > Resources 14
  • 14. Java Memory Tools > JPS • Getting the Process ID (PID) > Jstat • jstat -gcutil <pid> 250 7
  • 15. Looking at the Heap > %JAVA_HOME%/bin/jmap – histo:live <pid> • Looking at all the “live” objects > %JAVA_HOME%/bin/jmap – histo <pid> • Looking at all objects > The difference between is the list of unreachable objects
  • 16. Taking a Heap Dump > %JAVA_HOME%/bin/jmap – dump:live,file=heap.out,format=b <pid> • Dumps the Heap to a file > JConsole
  • 17. JHat > %JAVA_HOME%/bin/jhat <filename> • Starts a web server to investigate the heap > Queries • Show instance count for all classes • Show Heap Histogram • Show Finalizer • Use the Execute Object Query Language (OQL) • select s from java.lang.String s where s.count >=100
  • 18. JMX – Looking at Flags
  • 19. MAT – Memory Analyzer Tool
  • 20. VisualVM > Open Source All-in-One Java Troubleshooting tool > https://visualvm.dev.java.net/ 21
  • 21. Visualgc > visualgc <pid> > Visual Garbage Collection Monitoring • a graphical tool for monitoring the HotSpot Garbage Collector, Compiler, and class loader. It can monitor both local and remote JVMs.
  • 22. 23 DEMO
  • 23. Agenda > Java Memory Management > Memory Management Tools • Command-line Tools • VisualVM > Btrace > Summary > Resources 24
  • 24. BTrace > dynamically bytecode instrumenting (BCI) • read / not write • probe-based • observe running Java applications > Integration with DTrace on Solaris > http://btrace.dev.java.net > visualvm plugin 25
  • 25. BTrace Tools 26 Target JVM BTrace Agent VisualVM + BTrace Plugin BTrace command-line
  • 26. BTrace Terminology > Probe Point • “location” or “event” at which tracing statements are executed > Trace Actions • statements which are executed whenever a probe fires > Action Methods • static methods which define a trace 27
  • 27. Probes and Actions > Probe Targets • method entry / exit • line number • exceptions • return from method • exception throw (before) • synchronization entry / exit > Actions • static methods in trace class 28
  • 28. BTrace Restrictions > no new objects > no new arrays > no exceptions > no outer, inner, nested or local classes > no synchronization blocks > no loops > no interfaces 29
  • 29. Tracing > Annotations • com.sun.btrace.annotations • @BTrace • denotes a btrace class > Probe Points • @OnMethod • @OnTimer • @OnEvent • @OnExit 30
  • 30. Simple Example: Looking for Object Size 31
  • 31. Simple Example: Looking for Object Size 32
  • 32. BTrace Run Options > BTrace Command-Line • btrace • runs btrace tool (and compiles btrace script if necessary) • btracec • btrace script compiler • btracer • convenience script to start java project with tracing enabled at application startup > VisualVM + BTrace Plugin 33
  • 33. Testing the Probe > Start Target Application • java -jar java2demo.jar > Get the PID • jps > Inject Probe • btrace <pid> Sizeof.java 34
  • 34. BTrace + JMX Script 35
  • 35. BTrace + jstat 36
  • 36. ThreadLocal 37
  • 37. BTrace Events 38
  • 38. Checking Synchronization Entry / Exits 39
  • 39. Tracing Opportunities > Thread Monitors > Socket / Web Services > Object Creation and Size > File Access 40
  • 40. 41 DEMO
  • 41. Known Solutions with BTrace > Terracotta • concurrency issue > Hibernate / Atlassian Issue • thread / session management issue • http://jira.atlassian.com/browse/CONF-12201 42
  • 42. Summary > jmap, jhat, jconsole, jstat • tools already in the JDK bin directory > VisualVM • swiss army knife for JVM process monitoring and tracing > BTrace • dynamic tracing tool 43
  • 43. Resources > Performance and Troubleshooting • http://java.sun.com/performance/reference/ whitepapers/6_performance.html • http://java.sun.com/javase/6/webnotes/trouble/TSG- VM/html/docinfo.html > VisualVM • https://visualvm.dev.java.net/ > BTrace • https://btrace.dev.java.net/ 44
  • 44. Ken Sipe http://kensipe.blogspot.com twitter: @kensipe