Copyright © 2014 Splunk Inc.
Splunking the JVM
Damien Dallimore
Dev Evangelist , CSO Office @ Splunk

Disclaimer
During the course of this presentation, we may make forward looking statements regarding future events or the
expected performance of the company. We caution you that such statements reflect our current expectations and
estimates based on factors currently known to us and that actual events or results could differ materially. For important
factors that may cause actual results to differ from those contained in our forward-looking statements, please review
our filings with the SEC. The forward-looking statements made in the this presentation are being made as of the time
and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or
accurate information. We do not assume any obligation to update any forward looking statements we may make. In
addition, any information about our roadmap outlines our general product direction and is subject to change at any
time without notice. It is for informational purposes only and shall not, be incorporated into any contract or other
commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include
any such feature or functionality in a future release.
2

3
From Middle Earth
Make Splunk Apps & Add-ons
JVM background

4

apps.splunk.com
github.com/damiendallimore
5

What is this JVM thing ?

JVM = Java Virtual Machine
First appeared in the early 90’s
Now the dominant runtime for enterprise
applications
Application Servers Enterprise Service Buses Databases
NoSQL Distributed Big Data Web Servers
Directory Servers Search Engines Build Systems
Gaming Platforms Trading Systems Reservation Systems
Core Banking Messaging Infrastructure Proprietary Systems
7

8

Many JVM Variants
Oracle Hotspot
OpenJDK
Oracle JRockit
9
IBM J9
Azul Zing
More or less the same. OpenJDK
is reference impl for Java 7/8
Getting merged into Hotspot
Commercial alternatives

JVMs are here to stay
It’s not just about the “J”
Big Data frameworks
You should care about getting insights into your
JVM data
10

Getting at the data

Data Sources
Logs
Developers
JMX (Java Management Extensions)
Instrumentation Agents
Operating System
Distributed Communications
12

Log Data

Standard Log Files
Oct 21, 2013 4:42:15 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 1153 ms
Oct 21, 2013 4:42:15 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
Application logs that are part of the product
Developer logs for any code that was deployed
14
Written to local disk or network storage

Garbage Collector Logs
54.736: [Full GC 54.737:
[Tenured: 172798K->18092K(174784K), 2.3792658 secs] 257598K->18092K(259584K),
[Perm : 20476K->20476K(20480K)], 2.4715398 secs] [Times: user=0.56 sys=0.05, real=0.07 secs]
Generated by way of arguments passed to the JVM at startup
15
-verbose:gc
-Xloggc:/home/damien/jvm_logs/gc.log
-XX:+PrintGC
-XX:+PrintGCTimeStamps
-XX:+PrintGCDetails

SplunkJavaLogging
Sometimes you can’t write to file or deploy a UF
Appenders for Java Util Logging , Log4J , Logback
Simply add a logging appender to your logging configuration file
16

LogBack Appender Example
17

Code
18

Best practice semantic format
2012-08-07 15:54:06:644+1200 name="Failed Login" event_id=”12345" app="myapp" user="jane”
19

Better Exception Logging
20

Easier to work with in Splunk
21

Splunk SDK for Java
Use the SDK from any JVM Language , Java / Groovy / Scala etc….
Send log events via REST , UDP or TCP directly to Splunk from your code
Spring Integration Adaptors available on Github
22
SDK available from dev.splunk.com

Using Java SDK
23

Using Spring
24

JMX

JMX = Java Management Extensions
Monitor JVM via MBean attributes , operations and notifications
JVM MBeans
Vendor MBeans
Custom Coded MBeans
26

Browse MBeans with JConsole
27

Getting this data into Splunk
Runs on all supported Splunk platforms
Works with all main JVM variants
100% Free and Open Source
28

Simple to Configure
29

Many Connectivity Options
Let’s look at this Splunk App in action
30

31
This is great for monitoring a
single JVM……

But what if we have dozens , hundreds , even thousands ?
32

Multiple jmxserver elements per configuration file
33

Multiple configuration files per Splunk Instance
34

But stacking vertically will only get you so far
35

Go horizontal to achieve real scale
36
Indexer Cluster
UFs with JMX Mod Input
Monitored JVMs

In my experience , the data sources shown so far will be
thorough enough for most of your JVM monitoring use cases.
But what if you want to go deeper ?
37

Instrumentation
Agents

SplunkJavaAgent
Dynamic Byte Code Injection Agent for JVMs
No need to change any code to get APM level insights
This is all you pass to the JVM at startup :
-javaagent:splunkagent.jar
39

What raw metrics can you get ?
40
Bytecode Injection
• class loading
• method execution
• method timings
• method call stack
• caught & uncaught exceptions
JMX
• running in local in-memory mode
Binary HPROF Memory dumps decoded in text
• no more static hprof dumps and then loading into another Mem Analysis tool

Simple Configuration File
41

42

Raw events streamed into Splunk
43

Search with Splunk
44

Reveal APM insights
45

Operating System

The JVM executes as a process
Let’s see what the host Operating System can tell us : top
External programs that provide additional JVM insights : jstat
Index this data in Splunk and correlate
47

top
48

49

jstat
50

51

Distributed
Communications

Valuable data flows in & out of JVMs
53
Messaging
Capture packets off the wire

Build a Splunk App !

Opportunity
Reuse some of the data collection approaches in this presentation
and build out a specific JVM app
55
Publish it on apps.splunk.com
Make some money
Generate publicity for yourself / your company
Generate sales leads
Sell more Splunk licenses if you are also a partner
Do it because you love community collaboration
Become my beer buddy for life

Some JVM App Ideas
Splunk for JBoss
Splunk for Tomcat
Splunk for Solr
Splunk for Cassandra
Splunk for “Apache Project”
I am here to help you
56

THANK YOU
@damiendallimore
ddallimore@splunk.com

Questions ?