Splunking the JVM (Java Virtual Machine)


Published on

Splunk for JMX App overview (configuration, deployment, tips and tricks). Developing JMX logic in your application. Splunking other JVM logs and profiling traces. The JVM application landscape and why it's such a rich source of Splunkable machine data. Developing new Splunkbase apps to leverage Splunk for JMX.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Open JDK – Browser plugin and web start only with Oracle Java
  • Splunking the JVM (Java Virtual Machine)

    1. 1. Splunking the Java Virtual Machine(JVM) Presented by Damien Dallimore Developer Evangelist at SplunkCopyright © 2012 Splunk Inc.
    2. 2. About me• Developer Evangelist at Splunk since July 2012 • http://dev.splunk.com • http://splunk-base.splunk.com • Slides available for my “Using the Java SDK” session• Splunk Community Member • Splunk for JMX • SplunkJavaLogging • SplunkBase Answers• Splunk Architect and Administrator• Coder, hacker, architect of Enterprise Java solutions around the globe in many different industries(aviation, core banking, card payments etc…)• If Splunk had been there at the start of my career I would have a lot more hair today  2
    3. 3. Agenda• The JVM Landscape• JVM Machine Data• Splunk for JMX• Community Projects – call to arms• Questions (feel free to yell out at any time also)
    4. 4. The JVM Landscape
    5. 5. What is this JVM thing ?• Circa 1991, Dr. James Gosling at Sun started developing a technology for next generation smart devices/appliances• “Green” became “Oak” which became “Java”• Java 1.0 first appeared in January 1996.• The JVM is a virtual machine that runs programs that are compiled into Java bytecode• Available for many hardware and software platforms• 17 years later , the JVM has evolved from a consumer device technology, to a browser oriented technology with the explosion of the web , to now becoming deeply rooted in the enterprise software landscape on the server side and in the cloud 5
    6. 6. 17 years later • Oracle took ownership of Java from Sun in January 2010 • The Java Community Process(JCP) is the forum where members develop specifications for Java technology • Java Specification Requests(JSR) get submitted for new features, are reviewed and then voted on by the JCP ExecutiveApplication Servers Enterprise Service Buses Databases committee.NoSQL Distributed Big Data Web Servers • Editions • Embedded Java, Java ME , Java SE , Java EEDirectory Servers Search Engines Build Systems • Current Version is Java 7 (Dolphin) • Java 8 scheduled for 2013Gaming Platforms Trading Systems Reservation SystemsCore Banking Messaging Infrastructure Proprietary Systems 6
    7. 7. JVM Variants• Oracle Hotspot (formerly SUN) • the primary reference JVM implementation• Oracle JRockit (formerly BEA) • free since May 2011 • code base currently being merged with Hotspot, ETA ~JDK 8• Open JDK • SUN open sourced Hotspot and the Java class library in 2006 • Slight differences with Oracle Java still • OpenJDK is the official Java SE7 Reference Implementation• J9 • IBM’s JVM for AIX, Linux, MVS, OS/400, Pocket PC, z/OS• Azul Systems Zing • based on HotSpot • supports memory heaps up to 512 GB without GC pauses and is able to grow and shrink the heap based on loadhttp://en.wikipedia.org/wiki/List_of_Java_virtual_machines 7
    8. 8. The JVM has a healthy future• Hotspot / JRockit code merge creating a best of breed JVM, Oracle to contribute this to OpenJDK• OpenJDK is thriving, Oracle are contributing and being good stewards of Java (despite initial skepticism)• Proliferation of alternative JVM languages that can all co-habitate in the JVM and new features in Java 8 to further enhance this multi language platform • Scala • Groovy • Clojure• The JVM is evolving organically with the shifting tides of Enterprise software, it isn’t about the “J” anymore.• From the clustered Application Server domination of the 00’s we now see an explosion of Big Data products running in massively distributed environments on commodity hardware or in the cloud • Apache Hadoop family (MapReduce, Hive, Hbase, Cassandra, HDFS) 8
    9. 9. What is running in JVMs ? 9
    10. 10. JVM “Fanboi” Speaking of Java as a language as opposedFanboi Dr. Gosling to the JVM platform, James Gosling, the Father of Java, said "Most people talk about Java the language, and this may sound odd coming from me, but I could hardly care less." He went on to explain, "What I really care about is the Java Virtual Machine as a concept, because that is the thing that ties it all together." 10
    11. 11. JVM Machine Data
    12. 12. JVM Machine Data• The JVM footprint cross cuts the data centre and represents a massive source of valuable machine data• Large scale Application/Web Server clusters• Hadoop & Cassandra Node topologies in the 100’s and in some cases 1000’sCustom Developed Code WAR file CORRELATE JMX, Developer Logs, Splunk Java SDK, SplunkJavaLogging Application Code Tomcat JMX, Application Logs JVM Hotspot JMX, SNMP, HPROF,GC Logs, Custom Agents, Usage Tracker Operating System Linux JVM process OS resource metrics 12
    13. 13. Application & Developer Logs Splunk Indexer • Application logs • default logs that are part of the product Splunk Universal Forwarder • Developer logs • any custom code created and deployed to the application that has it’s own loggingDeveloped Code • Written to local disk or a mounted network M onitor Log Files/ Directorys Application volume JVM • Monitor with a Splunk UF OS 13
    14. 14. Splunk Java SDK / SplunkJavaLogging Splunk Indexer HTTP$REST$/$TCP$/$UDP • Alternative to writing to log file orDeveloped Code needing to deploy a Splunk Universal Forwarder Application • Use the Splunk Java SDK to input events directly to Splunk via HTTP Rest. JVM • Use SplunkJavaLogging to input events directly to Splunk using custom logging appenders. OS • Come to my “Using the Java SDK” session for more on this !! 14
    15. 15. JVM Process OS Metrics Splunk Indexer • By JVM Process ID : Process State, Memory, CPU, Disk Usage, Disk Splunk for Unix or Linux I/O, Network I/O, File Descriptor Usage. • Some OS metrics also exposed via JMXPoll output from • Splunk for Unix and Linux M onitor Log Files &commands Directorys • Splunk for Windows • Correlate this OS data across your JVM and Developed Code Application events ie: your JVM may have hung Application because of CPU starvation caused by some other process thrashing JVM OS 15
    16. 16. Garbage Collection logs Splunk Indexer • Extended Hotspot JVM options -verbose:gc -Xloggc:/home/damien/jvm_logs/gc.log Splunk Universal Forwarder -XX:+PrintGC -XX:+PrintGCTimeStamps -XX:+PrintGCDetailsDeveloped Code • The log is written to at Garbage Collection time Application • Be careful , can affect performance • Need to perform field extractions in Splunk JVM Monitor GC Log Files • GC metrics also available via JMX OS 54.736: [Full GC 54.737: [Tenured: 172798K->18092K(174784K), 2.3792658 secs] 257598K->18092K(259584K), [Perm : 20476K->20476K(20480K)], 2.4715398 secs] [Times: user=0.56 sys=0.05, real=0.07 secs] 16
    17. 17. Custom JVMTI Agents (Advanced) Splunk Indexer • Java Virtual Machine Tool Interface • Write custom agents that get injected into REST/TCP/UDP the natively running JVM • Dynamically inspect the state of applications Splunk Universal running in the JVMDeveloped Code Forwarder • Profiling, debugging, monitoring, thread/me Application mory analysis…the JVMTI Interface has extensive coverage JVM Monitor Agent Log Files • As you write the agent code , the data output can be file based or over the OS network 17
    18. 18. Usage Tracker for Oracle JVMs • Enable via a JVM system property and a config file Splunk Indexer -Dcom.oracle.usagetracker.config.file=/path/usagetracker.properties • Output to CSV file or over UDP VM start, UDP Fri Oct 22 14:13:03 BST 2010, examplehost/, Splunk Universal AppName, Forwarder /path/to/jre,Developed Code 1.7.0, 19.0-b09, Application Oracle Corporation, Oracle Corporation, Linux, JVM Usage Tracker Log Files i386, 2.6.29.x86_64, -Xmx128m, OS /opt/programs, user.home=/home/username foo.bar=null • All these metrics also available via JMX 18
    19. 19. SNMP Splunk Indexer • The JVM SNMP Agent provides a single MIB that exposes the JVM’s Management and Monitoring API http://docs.oracle.com/javase/1.5.0/docs/guide/management/JVM-MANAGEMENT-MIB.mib Splunk UniversalDeveloped Code Forwarder • Setup the JVM (just the basic settings shown) SNMP% Objects% Polled Application JVM MIB Open a UDP Port : -Dcom.sun.management.snmp.port=9004 JVM Configure the ACL : $JAVA_HOME/jre/lib/management/snmp.acl OS • Traps can be caught locally to file and monitored SNMP% Traps% wri6en% file to% • A scripted input on the Splunk UF can poll the JVM SNMP Objects pysnmp python module : http://pysnmp.sourceforge.net snmptrapd UDP:162 snmpget command : http://www.net-snmp.org/docs/man/snmpget.html There is a nice example of this on SplunkBase 19
    20. 20. HPROF Profiling Dumps Splunk Indexer • Binary JVM dumps that allow for deeper JVM resource inspection • Typical use case is diagnosing memory issues after JVM crashes with java.lang.OutOfMemoryError Splunk Universal • Binary file is usually batch loaded into a third party Forwarder memory analysis tool like Eclipse MAT • Generate a heap dump on demand via JMX M onitor and decode into • Or tell the JVM to generate a heap dump under certainDeveloped Code textual key=value pairs conditions : • -XX:HeapDumpPath=./java_pid<pid>.hprof Application • -XX:-HeapDumpOnOutOfMemoryError JVM • But what if we could Splunk this awesome source of information, this could be really useful in dev/test ! OS Binary HPROF dump file Warning : heap dumping is an expensive operation as a full GC gets performed 20
    21. 21. Splunk HPROF Decoder• A scripted input that monitors for HPROF file dumps , reads the binary file in and rolls it out into key=value format for Splunking• Deploy the Scripted input to a Universal Forwarder• Use Splunk for JMX to periodically trigger an HPROF dump via a JMX operation• Splunk Universal Splunk is now a JVM Heap Profiling utility Forwarder• Diagnose Heap issues before they hit production M onitor and decode into• Trigger HPRO F file generation Splunk for JMX can tell you that the Heap is growing via a JM X operation textual key=value pairs• This will tell you what is causing the growth JVM Binary HPRO F dump file 21
    22. 22. Splunk Heap Memory Analysis 22
    23. 23. JMX (Java Management Extensions) Splunk Indexer • Manage and Monitor the JVM and Application via exposed MBeans • JVM MBeans (java.lang domain) • Vendor MBeans (most vendors ship their products with extensive MBean coverage) • Custom Coded MBeans (whatever your devs wish toDeveloped Code JMX code) Splunk Universal • MBeans expose attributes, operations and Application Forwarder notifications to give you a powerfully dynamic JVM insight into the runtime state of the JVM and your application. OS • Add Splunk to the mix for historical and realtime operational visibility, pro-active issue detection etc.. • Splunk for JMX app on SplunkBase 23
    24. 24. JMX vs SNMPJMX• Open and easily extensible• Developers can simply create new MBeans• Vendor products(JBoss, Cassandra, Hadoop etc..) ship with thorough MBean coverage, not MIBsSNMP• The built-in SNMP agent of the JVM is not extensible.• You will not be able to use it in order to expose your own custom MIB• If you do want to expose your own MIB, you’d have to create a custom agent 24
    25. 25. Putting it all together, JVM Splunking Nirvana Distributed Search Splunk Indexer Cluster Auto Load Balanced REST/TCP/UDP JMX Developed Code Logs JMX Splunk Forwarder Application Logs JMX JVM HPROF OS* Metrics/Logs OS 25
    26. 26. Splunk for JMX
    27. 27. Splunk for JMX• Connect to any local or remote JVMs JMX server, Hotspot/JRockit/IBM J9• Query any MBean running on that server• Extract any MBean attributes (simple, composite or tabular)• Invoke MBean operations• Write attributes and operation results out in a default key/value format, or plugin your own custom format, for SPLUNK indexing and searching• Transport events over STD OUT(default), TCP, Syslog, Splunk REST endpoint or direct to file.• Declare clusters of JVMs for larger scale JVM deployments• Runs on *Nix and Windows• Out of the box dashboards for common JVM MBeans• Freely available from SplunkBase, all source code is on GitHub 27
    28. 28. Connectivity OptionsRemote JMX interface• rmi (JSR160 Standard Implementation and MX4Js JSR160 Implementation)• iiop (JSR160 Standard Implementation and MX4Js JSR160 Implementation)Direct Process attachment• Connect directly to a locally running JVM processMX4J HTTP connectors (requires MX4J in the target JVM also)• soap , soap+sssl• hessian, hessian+ssl• burlap, burlap+ssl 28
    29. 29. Setup and ConfigurationThe main goal of the app was to make it as simple and intuitive as possible to connectto your JVMs and start Splunking JMX data• Enable your target JVM’s remote JMX interface , test connectivity with JConsole• Install Splunk for JMX • Set your SPLUNK_HOME , JAVA_HOME environment variables, JRE 6+ required • Extract Splunk for JMX tarball to SPLUNK_HOME/etc/apps • Restart Splunk • At the setup screen, choose a scripted input for your platform (Nix / Windows)• Setup your JMX configuration file • The default config.xml file is pre configured for common JVM MBeans • Browse your JVM (using JConsole) for other MBeans that you wish to poll and configure these • You can have as many config files as you require, and you might set these up to fire off at different scheduled frequencies 29
    30. 30. Configuration Examples - Simple• MBean Object name format “domain:key=value,key2=value2”• * and ? wildcards are supported in the Mbean name Around 25KBytes per dump on Hotspot JVMs 30
    31. 31. Configuration Examples - Clusters • Define clusters of JVM’s that share the same MBean definitions • Note , in these examples, for brevity I am using “dumpAllAttributes” , but in production you’d want to pick and choose specific MBean attributes you are interested in, and perhaps split definitions over multiple files run at varying frequencies 31
    32. 32. Configuration Examples - Operations • Invoke JMX operations that return a value or simply perform some action on the target JVM • Operation definitions can take parameters Use Case 1 : your developers might code a JMX operation that returns a CSV or JSON formatted snapshot of some metrics for Splunking Use Case 2 : dynamically trigger HPROF dumps. The “com.sun.management:type=HotSpotDiagnostic” Mbean exposes a “dumpHeap” operation 32
    33. 33. Configuration Examples - Connecting • IP Address with credentials • Hostname • Static Process ID • Process ID lookup from file • Process ID lookup from command output • Raw JMX Service URL • MX4J HTTP Connector 33
    34. 34. Custom Formatters/Transports• The Splunk for JMX configuration is user extensible• You can code and configure your own Formatters and TransportsFormatters• Takes the raw MBean polled output and formats it for Splunking• A Java implementation of the "com.dtdsoftware.splunk.formatter.Formatter" interface• If the optional formatter declaration is omitted, then the default formatter will be usedTransports• Takes the formatted output and transports it to a destination• A Java implementation of the "com.dtdsoftware.splunk.transport.Transport" interface• If the optional transport declaration is omitted, then the default transport(STD out) will be used 34
    35. 35. Formatter Examples 35
    36. 36. Transport Examples 36
    37. 37. Deployment Architectures 1 • Simplest scenario • Monolithic Splunk installation • Splunk for JMX polling 1 or more remote/local JVMs via the remote JMX interface • There is support for many target JVM’s in the configuration schema but to really scale out, you need a more advanced Splunk architecture 37
    38. 38. Deployment Architectures 2 Load Balancer • Run Splunk UF locally with target JVM.Can connect use remote JMX interface orSplunk Search HeadPool direct process attachment. • Each tier scales out horizontally. • Can overcome firewall issues that are Splunk Indexer sometimes inherent with Java RMI Cluster • Deploy Splunk for JMX components and configurations with Splunk Deployment Server, Puppet or Chef. Splunk UF running locally with target JVM 38
    39. 39. Community Projects – call to arms !!
    40. 40. Remember this slide ? 40
    41. 41. SplunkBase JVM Apps• I’ve already started on some, but I can’t do it all myself !• You can use Splunk for JMX as the “kernel” upon which to build Splunk for Tomcat, Splunk for JBoss, Splunk for Mule etc..• I have found that with most of the JVM apps that I have looked at or been asked to build a Splunk app for, that most of the useful data is in the JMX metrics and operations• Any this can of course be augmented with any useful log data• Build Simple/Advanced XML dashboards• Bundle up the app and post it on Splunkbase, share with the community and perhaps someone else will create an app that you can use too• Note , you are publishing a common app so you can’t take into account any custom developer code, just the metrics and logs that are inherent to the core JVM app 41
    42. 42. Contact DetailsAlways more than happy to be contacted forquestions, feedback, collaborations, ideas that will change theworld etc…Email : ddallimore@splunk.comSplunkBase: damiendGithub: damiendallimoreTwitter : @damiendallimoreBlog : http://blogs.splunk.com/devSplunk Dev Platform Team : devinfo@splunk.com 42
    43. 43. LinksSplunk for JMX: http://splunk-base.splunk.com/apps/25505/splunk-for-jmxSplunkJavaLogging: https://github.com/damiendallimore/SplunkJavaLoggingSplunk Java SDK: http://dev.splunk.com/view/java-sdk/SP-CAAAECNOracle Java: http://www.oracle.com/us/technologies/java/overview/index.htmlOpen JDK : http://openjdk.java.net/JMX : http://www.oracle.com/technetwork/java/javase/tech/javamanagement-140525.htmlAzul Zing : http://www.azulsystems.com/products/zing/whatisitJVMTI : http://docs.oracle.com/javase/6/docs/technotes/guides/jvmti/Usage Tracker : http://docs.oracle.com/javase/products/usagetracker.htmlUsage Tracker w/ Splunk : http://javalandtales.blogspot.co.uk/#!/2012/05/using-java-usage-tracker-feature-with.html 43
    44. 44. Thanks for coming !