Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Splunk Conf 2014 - Splunking the Java Virtual Machine


Published on

Presentation from Splunk Conf 14

Published in: Software
  • Login to see the comments

Splunk Conf 2014 - Splunking the Java Virtual Machine

  1. 1. Copyright © 2014 Splunk Inc. Splunking the JVM Damien Dallimore Dev Evangelist , CSO Office @ Splunk
  2. 2. Disclaimer During the course of this presentation, we may make forward looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in the this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release. 2
  3. 3. 3 From Middle Earth Make Splunk Apps & Add-ons JVM background
  4. 4. 4
  5. 5. 5
  6. 6. What is this JVM thing ?
  7. 7. JVM = Java Virtual Machine First appeared in the early 90’s Now the dominant runtime for enterprise applications Application Servers Enterprise Service Buses Databases NoSQL Distributed Big Data Web Servers Directory Servers Search Engines Build Systems Gaming Platforms Trading Systems Reservation Systems Core Banking Messaging Infrastructure Proprietary Systems 7
  8. 8. 8
  9. 9. Many JVM Variants Oracle Hotspot OpenJDK Oracle JRockit 9 IBM J9 Azul Zing More or less the same. OpenJDK is reference impl for Java 7/8 Getting merged into Hotspot Commercial alternatives
  10. 10. JVMs are here to stay It’s not just about the “J” Big Data frameworks You should care about getting insights into your JVM data 10
  11. 11. Getting at the data
  12. 12. Data Sources Logs Developers JMX (Java Management Extensions) Instrumentation Agents Operating System Distributed Communications 12
  13. 13. Log Data
  14. 14. Standard Log Files Oct 21, 2013 4:42:15 PM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 1153 ms Oct 21, 2013 4:42:15 PM org.apache.catalina.core.StandardService startInternal INFO: Starting service Catalina Application logs that are part of the product Developer logs for any code that was deployed 14 Written to local disk or network storage
  15. 15. Garbage Collector Logs 54.736: [Full GC 54.737: [Tenured: 172798K->18092K(174784K), 2.3792658 secs] 257598K->18092K(259584K), [Perm : 20476K->20476K(20480K)], 2.4715398 secs] [Times: user=0.56 sys=0.05, real=0.07 secs] Generated by way of arguments passed to the JVM at startup 15 -verbose:gc -Xloggc:/home/damien/jvm_logs/gc.log -XX:+PrintGC -XX:+PrintGCTimeStamps -XX:+PrintGCDetails
  16. 16. SplunkJavaLogging Sometimes you can’t write to file or deploy a UF Appenders for Java Util Logging , Log4J , Logback Simply add a logging appender to your logging configuration file 16
  17. 17. LogBack Appender Example 17
  18. 18. Code 18
  19. 19. Best practice semantic format 2012-08-07 15:54:06:644+1200 name="Failed Login" event_id=”12345" app="myapp" user="jane” 19
  20. 20. Better Exception Logging 20
  21. 21. Easier to work with in Splunk 21
  22. 22. Splunk SDK for Java Use the SDK from any JVM Language , Java / Groovy / Scala etc…. Send log events via REST , UDP or TCP directly to Splunk from your code Spring Integration Adaptors available on Github 22 SDK available from
  23. 23. Using Java SDK 23
  24. 24. Using Spring 24
  25. 25. JMX
  26. 26. JMX = Java Management Extensions Monitor JVM via MBean attributes , operations and notifications JVM MBeans Vendor MBeans Custom Coded MBeans 26
  27. 27. Browse MBeans with JConsole 27
  28. 28. Getting this data into Splunk Runs on all supported Splunk platforms Works with all main JVM variants 100% Free and Open Source 28
  29. 29. Simple to Configure 29
  30. 30. Many Connectivity Options Let’s look at this Splunk App in action 30
  31. 31. 31 This is great for monitoring a single JVM……
  32. 32. But what if we have dozens , hundreds , even thousands ? 32
  33. 33. Multiple jmxserver elements per configuration file 33
  34. 34. Multiple configuration files per Splunk Instance 34
  35. 35. But stacking vertically will only get you so far 35
  36. 36. Go horizontal to achieve real scale 36 Indexer Cluster UFs with JMX Mod Input Monitored JVMs
  37. 37. In my experience , the data sources shown so far will be thorough enough for most of your JVM monitoring use cases. But what if you want to go deeper ? 37
  38. 38. Instrumentation Agents
  39. 39. SplunkJavaAgent Dynamic Byte Code Injection Agent for JVMs No need to change any code to get APM level insights This is all you pass to the JVM at startup : -javaagent:splunkagent.jar 39
  40. 40. What raw metrics can you get ? 40 Bytecode Injection • class loading • method execution • method timings • method call stack • caught & uncaught exceptions JMX • running in local in-memory mode Binary HPROF Memory dumps decoded in text • no more static hprof dumps and then loading into another Mem Analysis tool
  41. 41. Simple Configuration File 41
  42. 42. 42
  43. 43. Raw events streamed into Splunk 43
  44. 44. Search with Splunk 44
  45. 45. Reveal APM insights 45
  46. 46. Operating System
  47. 47. The JVM executes as a process Let’s see what the host Operating System can tell us : top External programs that provide additional JVM insights : jstat Index this data in Splunk and correlate 47
  48. 48. top 48
  49. 49. 49
  50. 50. jstat 50
  51. 51. 51
  52. 52. Distributed Communications
  53. 53. Valuable data flows in & out of JVMs 53 Messaging Capture packets off the wire
  54. 54. Build a Splunk App !
  55. 55. Opportunity Reuse some of the data collection approaches in this presentation and build out a specific JVM app 55 Publish it on Make some money Generate publicity for yourself / your company Generate sales leads Sell more Splunk licenses if you are also a partner Do it because you love community collaboration Become my beer buddy for life
  56. 56. Some JVM App Ideas Splunk for JBoss Splunk for Tomcat Splunk for Solr Splunk for Cassandra Splunk for “Apache Project” I am here to help you 56
  57. 57. THANK YOU @damiendallimore
  58. 58. Questions ?