More Related Content


Similar to Splunk Conf 2014 - Splunking the Java Virtual Machine(20)


Splunk Conf 2014 - Splunking the Java Virtual Machine

  1. Copyright © 2014 Splunk Inc. Splunking the JVM Damien Dallimore Dev Evangelist , CSO Office @ Splunk
  2. Disclaimer During the course of this presentation, we may make forward looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in the this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release. 2
  3. 3 From Middle Earth Make Splunk Apps & Add-ons JVM background
  4. 4
  5. 5
  6. What is this JVM thing ?
  7. JVM = Java Virtual Machine First appeared in the early 90’s Now the dominant runtime for enterprise applications Application Servers Enterprise Service Buses Databases NoSQL Distributed Big Data Web Servers Directory Servers Search Engines Build Systems Gaming Platforms Trading Systems Reservation Systems Core Banking Messaging Infrastructure Proprietary Systems 7
  8. 8
  9. Many JVM Variants Oracle Hotspot OpenJDK Oracle JRockit 9 IBM J9 Azul Zing More or less the same. OpenJDK is reference impl for Java 7/8 Getting merged into Hotspot Commercial alternatives
  10. JVMs are here to stay It’s not just about the “J” Big Data frameworks You should care about getting insights into your JVM data 10
  11. Getting at the data
  12. Data Sources Logs Developers JMX (Java Management Extensions) Instrumentation Agents Operating System Distributed Communications 12
  13. Log Data
  14. Standard Log Files Oct 21, 2013 4:42:15 PM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 1153 ms Oct 21, 2013 4:42:15 PM org.apache.catalina.core.StandardService startInternal INFO: Starting service Catalina Application logs that are part of the product Developer logs for any code that was deployed 14 Written to local disk or network storage
  15. Garbage Collector Logs 54.736: [Full GC 54.737: [Tenured: 172798K->18092K(174784K), 2.3792658 secs] 257598K->18092K(259584K), [Perm : 20476K->20476K(20480K)], 2.4715398 secs] [Times: user=0.56 sys=0.05, real=0.07 secs] Generated by way of arguments passed to the JVM at startup 15 -verbose:gc -Xloggc:/home/damien/jvm_logs/gc.log -XX:+PrintGC -XX:+PrintGCTimeStamps -XX:+PrintGCDetails
  16. SplunkJavaLogging Sometimes you can’t write to file or deploy a UF Appenders for Java Util Logging , Log4J , Logback Simply add a logging appender to your logging configuration file 16
  17. LogBack Appender Example 17
  18. Code 18
  19. Best practice semantic format 2012-08-07 15:54:06:644+1200 name="Failed Login" event_id=”12345" app="myapp" user="jane” 19
  20. Better Exception Logging 20
  21. Easier to work with in Splunk 21
  22. Splunk SDK for Java Use the SDK from any JVM Language , Java / Groovy / Scala etc…. Send log events via REST , UDP or TCP directly to Splunk from your code Spring Integration Adaptors available on Github 22 SDK available from
  23. Using Java SDK 23
  24. Using Spring 24
  25. JMX
  26. JMX = Java Management Extensions Monitor JVM via MBean attributes , operations and notifications JVM MBeans Vendor MBeans Custom Coded MBeans 26
  27. Browse MBeans with JConsole 27
  28. Getting this data into Splunk Runs on all supported Splunk platforms Works with all main JVM variants 100% Free and Open Source 28
  29. Simple to Configure 29
  30. Many Connectivity Options Let’s look at this Splunk App in action 30
  31. 31 This is great for monitoring a single JVM……
  32. But what if we have dozens , hundreds , even thousands ? 32
  33. Multiple jmxserver elements per configuration file 33
  34. Multiple configuration files per Splunk Instance 34
  35. But stacking vertically will only get you so far 35
  36. Go horizontal to achieve real scale 36 Indexer Cluster UFs with JMX Mod Input Monitored JVMs
  37. In my experience , the data sources shown so far will be thorough enough for most of your JVM monitoring use cases. But what if you want to go deeper ? 37
  38. Instrumentation Agents
  39. SplunkJavaAgent Dynamic Byte Code Injection Agent for JVMs No need to change any code to get APM level insights This is all you pass to the JVM at startup : -javaagent:splunkagent.jar 39
  40. What raw metrics can you get ? 40 Bytecode Injection • class loading • method execution • method timings • method call stack • caught & uncaught exceptions JMX • running in local in-memory mode Binary HPROF Memory dumps decoded in text • no more static hprof dumps and then loading into another Mem Analysis tool
  41. Simple Configuration File 41
  42. 42
  43. Raw events streamed into Splunk 43
  44. Search with Splunk 44
  45. Reveal APM insights 45
  46. Operating System
  47. The JVM executes as a process Let’s see what the host Operating System can tell us : top External programs that provide additional JVM insights : jstat Index this data in Splunk and correlate 47
  48. top 48
  49. 49
  50. jstat 50
  51. 51
  52. Distributed Communications
  53. Valuable data flows in & out of JVMs 53 Messaging Capture packets off the wire
  54. Build a Splunk App !
  55. Opportunity Reuse some of the data collection approaches in this presentation and build out a specific JVM app 55 Publish it on Make some money Generate publicity for yourself / your company Generate sales leads Sell more Splunk licenses if you are also a partner Do it because you love community collaboration Become my beer buddy for life
  56. Some JVM App Ideas Splunk for JBoss Splunk for Tomcat Splunk for Solr Splunk for Cassandra Splunk for “Apache Project” I am here to help you 56
  57. THANK YOU @damiendallimore
  58. Questions ?

Editor's Notes

  1. Legal disclaimer
  2. From Auckland Dev evang , ex customer 5th Conf Make Apps , Cut code Through enterprise Java background in many verticals , enterprise and non enterprise Java and other JVM langs
  3. Everything 100% open source use , reuse , whatever. Collaborate Community for support is best
  4. JVM is a runtime for code that gets compiled to bytecode and executed. Not natively. But can JIT. So performance is good now. Circa 1991, Dr. James Gosling at Sun started developing a technology for next generation smart devices/appliances “Green” became “Oak” which became “Java” Java 1.0 first appeared in January 1996. The JVM is a virtual machine that runs programs that are compiled into Java bytecode Available for many hardware and software platforms 17 years later , the JVM has evolved from a consumer device technology, to a browser oriented technology with the explosion of the web , to now becoming deeply rooted in the enterprise software landscape on the server side and in the cloud Overcame some limitatons on C/C++ Language changes , generics , annotations , lambda’s , streams , default methods GC , JIT , bytecode , write once run anywhere
  5. Ask crowd about JVM experience : Devs Operations Have no idea but might after this preso
  6. Jrockit and hotspot code being integrated Openjdk closely linked to hotspot , Oracle JDK has a few other proprietary add-ons above. Open JDK is the Java reference impl Zing is commercial
  7. Groovy , Scala , Clojure , JS , the JVM is evolving with the changing landscape of enterprise software From app servers in the 00’s to Big Data frameworks today Hadoop , Cassandra, 1000’s of nodes The JVM footprint cross cuts the data centre and represents a massive source of valuable machine data Large scale Application/Web Server clusters Hadoop & Cassandra Node topologies in the 100’s and in some cases 1000’s Segway into JVM Data
  8. Monitor with a UF = best practice
  9. JVM generated logs by way of options
  10. Try to use a UF if at all possible Utility classes for formatting log events Configurable in memory buffer to handle network outages
  11. TCP and REST Appenders
  12. SplunkLogEvent will log in best practive semantic format
  13. index=main sourcetype=log4j2 | makemv delim="," stacktrace_elements | table _time throwable_message throwable_class stacktrace_elements
  14. What if you can’t use logging appenders etc… Well , then you have to get down and dirty with your code !
  15. Majority of useful data for monitoring will come from this layer Open and easily extensible Developers can simply create new MBeans Vendor products(JBoss, Cassandra, Hadoop etc..) ship with thorough MBean coverage, not MIBs
  16. JConsole to Browse Mbeans Enable your target JVM’s remote JMX interface , test connectivity with JConsole
  17. for demo
  18. As many config files as you want Can make output CIM compliant
  19. Also MX4j connectivity
  20. Run Splunk UF locally with target JVM.Can connect use remote JMX interface or direct process attachment. Each tier scales out horizontally. Can overcome firewall issues that are sometimes inherent with Java RMI Deploy Splunk for JMX components and configurations with Splunk Deployment Server, Puppet or Chef. More than just jvm as bottleneck , also have stdin/out and Splunk indexing pipeline
  21. Appdynamics have an integration
  22. Hprof : Binary JVM dumps that allow for deeper JVM resource inspection Typical use case is diagnosing memory issues after JVM crashes with java.lang.OutOfMemoryError
  23. sourcetype=splunk_java_agent name=method* | transaction appName,appID,threadID,className,methodName startswith=method_entered endswith=method_exited | stats min(duration) as "Min Execution Time" max(duration) as "Max Execution Time" avg(duration) as "Avg Execution Time" count as "Total Calls" sum(duration) as "Cumulative Execution Time" by className,methodName
  24. Correlate this OS data across your JVM and Application events ie: your JVM may have hung because of CPU starvation caused by some other process thrashing Could even go lower if running virtual and get the VM/Hypervisor data
  25. Created a wrapper script to dynamically get the PID of the Java process
  26. Custom handlers for pre-processing and reformatting if you want
  27. Not going into details here , there are other sessions you can attend.
  28. ie: reuse the JMX Modular input