SlideShare a Scribd company logo
1 of 58
Copyright © 2014 Splunk Inc. 
Splunking the JVM 
Damien Dallimore 
Dev Evangelist , CSO Office @ Splunk
Disclaimer 
During the course of this presentation, we may make forward looking statements regarding future events or the 
expected performance of the company. We caution you that such statements reflect our current expectations and 
estimates based on factors currently known to us and that actual events or results could differ materially. For important 
factors that may cause actual results to differ from those contained in our forward-looking statements, please review 
our filings with the SEC. The forward-looking statements made in the this presentation are being made as of the time 
and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or 
accurate information. We do not assume any obligation to update any forward looking statements we may make. In 
addition, any information about our roadmap outlines our general product direction and is subject to change at any 
time without notice. It is for informational purposes only and shall not, be incorporated into any contract or other 
commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include 
any such feature or functionality in a future release. 
2
3 
From Middle Earth 
Make Splunk Apps & Add-ons 
JVM background
4
apps.splunk.com 
github.com/damiendallimore 
5
What is this JVM thing ?
JVM = Java Virtual Machine 
First appeared in the early 90’s 
Now the dominant runtime for enterprise 
applications 
Application Servers Enterprise Service Buses Databases 
NoSQL Distributed Big Data Web Servers 
Directory Servers Search Engines Build Systems 
Gaming Platforms Trading Systems Reservation Systems 
Core Banking Messaging Infrastructure Proprietary Systems 
7
8
Many JVM Variants 
Oracle Hotspot 
OpenJDK 
Oracle JRockit 
9 
IBM J9 
Azul Zing 
More or less the same. OpenJDK 
is reference impl for Java 7/8 
Getting merged into Hotspot 
Commercial alternatives
JVMs are here to stay 
It’s not just about the “J” 
Big Data frameworks 
You should care about getting insights into your 
JVM data 
10
Getting at the data
Data Sources 
Logs 
Developers 
JMX (Java Management Extensions) 
Instrumentation Agents 
Operating System 
Distributed Communications 
12
Log Data
Standard Log Files 
Oct 21, 2013 4:42:15 PM org.apache.catalina.startup.Catalina load 
INFO: Initialization processed in 1153 ms 
Oct 21, 2013 4:42:15 PM org.apache.catalina.core.StandardService startInternal 
INFO: Starting service Catalina 
Application logs that are part of the product 
Developer logs for any code that was deployed 
14 
Written to local disk or network storage
Garbage Collector Logs 
54.736: [Full GC 54.737: 
[Tenured: 172798K->18092K(174784K), 2.3792658 secs] 257598K->18092K(259584K), 
[Perm : 20476K->20476K(20480K)], 2.4715398 secs] [Times: user=0.56 sys=0.05, real=0.07 secs] 
Generated by way of arguments passed to the JVM at startup 
15 
-verbose:gc 
-Xloggc:/home/damien/jvm_logs/gc.log 
-XX:+PrintGC 
-XX:+PrintGCTimeStamps 
-XX:+PrintGCDetails
SplunkJavaLogging 
Sometimes you can’t write to file or deploy a UF 
Appenders for Java Util Logging , Log4J , Logback 
Simply add a logging appender to your logging configuration file 
16
LogBack Appender Example 
17
Code 
18
Best practice semantic format 
2012-08-07 15:54:06:644+1200 name="Failed Login" event_id=”12345" app="myapp" user="jane” 
19
Better Exception Logging 
20
Easier to work with in Splunk 
21
Splunk SDK for Java 
Use the SDK from any JVM Language , Java / Groovy / Scala etc…. 
Send log events via REST , UDP or TCP directly to Splunk from your code 
Spring Integration Adaptors available on Github 
22 
SDK available from dev.splunk.com
Using Java SDK 
23
Using Spring 
24
JMX
JMX = Java Management Extensions 
Monitor JVM via MBean attributes , operations and notifications 
JVM MBeans 
Vendor MBeans 
Custom Coded MBeans 
26
Browse MBeans with JConsole 
27
Getting this data into Splunk 
Runs on all supported Splunk platforms 
Works with all main JVM variants 
100% Free and Open Source 
28
Simple to Configure 
29
Many Connectivity Options 
Let’s look at this Splunk App in action 
30
31 
This is great for monitoring a 
single JVM……
But what if we have dozens , hundreds , even thousands ? 
32
Multiple jmxserver elements per configuration file 
33
Multiple configuration files per Splunk Instance 
34
But stacking vertically will only get you so far 
35
Go horizontal to achieve real scale 
36 
Indexer Cluster 
UFs with JMX Mod Input 
Monitored JVMs
In my experience , the data sources shown so far will be 
thorough enough for most of your JVM monitoring use cases. 
But what if you want to go deeper ? 
37
Instrumentation 
Agents
SplunkJavaAgent 
Dynamic Byte Code Injection Agent for JVMs 
No need to change any code to get APM level insights 
This is all you pass to the JVM at startup : 
-javaagent:splunkagent.jar 
39
What raw metrics can you get ? 
40 
Bytecode Injection 
• class loading 
• method execution 
• method timings 
• method call stack 
• caught & uncaught exceptions 
JMX 
• running in local in-memory mode 
Binary HPROF Memory dumps decoded in text 
• no more static hprof dumps and then loading into another Mem Analysis tool
Simple Configuration File 
41
42
Raw events streamed into Splunk 
43
Search with Splunk 
44
Reveal APM insights 
45
Operating System
The JVM executes as a process 
Let’s see what the host Operating System can tell us : top 
External programs that provide additional JVM insights : jstat 
Index this data in Splunk and correlate 
47
top 
48
49
jstat 
50
51
Distributed 
Communications
Valuable data flows in & out of JVMs 
53 
Messaging 
Capture packets off the wire
Build a Splunk App !
Opportunity 
Reuse some of the data collection approaches in this presentation 
and build out a specific JVM app 
55 
Publish it on apps.splunk.com 
Make some money 
Generate publicity for yourself / your company 
Generate sales leads 
Sell more Splunk licenses if you are also a partner 
Do it because you love community collaboration 
Become my beer buddy for life
Some JVM App Ideas 
Splunk for JBoss 
Splunk for Tomcat 
Splunk for Solr 
Splunk for Cassandra 
Splunk for “Apache Project” 
I am here to help you 
56
THANK YOU 
@damiendallimore 
ddallimore@splunk.com
Questions ?

More Related Content

What's hot

OSMC 2021 | inspectIT Ocelot: Dynamic OpenTelemetry Instrumentation at Runtime
OSMC 2021 | inspectIT Ocelot: Dynamic OpenTelemetry Instrumentation at RuntimeOSMC 2021 | inspectIT Ocelot: Dynamic OpenTelemetry Instrumentation at Runtime
OSMC 2021 | inspectIT Ocelot: Dynamic OpenTelemetry Instrumentation at RuntimeNETWAYS
 
Cloudify workshop at CCCEU 2014
Cloudify workshop at CCCEU 2014 Cloudify workshop at CCCEU 2014
Cloudify workshop at CCCEU 2014 Uri Cohen
 
OSMC 2021 || Open Source Application Performance Monitoring in the Enterprise
OSMC 2021 || Open Source Application Performance Monitoring in the EnterpriseOSMC 2021 || Open Source Application Performance Monitoring in the Enterprise
OSMC 2021 || Open Source Application Performance Monitoring in the EnterpriseNETWAYS
 
OSMC 2021 | Current State of Icinga
OSMC 2021 | Current State of IcingaOSMC 2021 | Current State of Icinga
OSMC 2021 | Current State of IcingaNETWAYS
 
Elk ruminating on logs
Elk ruminating on logsElk ruminating on logs
Elk ruminating on logsMathew Beane
 
Developing with the Go client for Apache Kafka
Developing with the Go client for Apache KafkaDeveloping with the Go client for Apache Kafka
Developing with the Go client for Apache KafkaJoe Stein
 
Performance Comparison of Streaming Big Data Platforms
Performance Comparison of Streaming Big Data PlatformsPerformance Comparison of Streaming Big Data Platforms
Performance Comparison of Streaming Big Data PlatformsDataWorks Summit/Hadoop Summit
 
Getting Pulsar Spinning_Addison Higham
Getting Pulsar Spinning_Addison HighamGetting Pulsar Spinning_Addison Higham
Getting Pulsar Spinning_Addison HighamStreamNative
 
Open source applied: Real-world uses
Open source applied: Real-world usesOpen source applied: Real-world uses
Open source applied: Real-world usesRogue Wave Software
 
How to build a Neutron Plugin (stadium edition)
How to build a Neutron Plugin (stadium edition)How to build a Neutron Plugin (stadium edition)
How to build a Neutron Plugin (stadium edition)Salvatore Orlando
 
Open source: Top issues in the top enterprise packages
Open source: Top issues in the top enterprise packagesOpen source: Top issues in the top enterprise packages
Open source: Top issues in the top enterprise packagesRogue Wave Software
 
Performance Testing using Real Browsers with JMeter & Webdriver
Performance Testing using Real Browsers with JMeter & WebdriverPerformance Testing using Real Browsers with JMeter & Webdriver
Performance Testing using Real Browsers with JMeter & WebdriverBlazeMeter
 
Configuration Management Tools on NX-OS
Configuration Management Tools on NX-OSConfiguration Management Tools on NX-OS
Configuration Management Tools on NX-OSCisco DevNet
 
Apache Spark the Hard Way: Challenges with Building an On-Prem Spark Analytic...
Apache Spark the Hard Way: Challenges with Building an On-Prem Spark Analytic...Apache Spark the Hard Way: Challenges with Building an On-Prem Spark Analytic...
Apache Spark the Hard Way: Challenges with Building an On-Prem Spark Analytic...Spark Summit
 
Interactive Analytics using Apache Spark
Interactive Analytics using Apache SparkInteractive Analytics using Apache Spark
Interactive Analytics using Apache SparkSachin Aggarwal
 
Helium makes Zeppelin fly!
Helium makes Zeppelin fly!Helium makes Zeppelin fly!
Helium makes Zeppelin fly!DataWorks Summit
 
Automation in Network Lifecycle Management - Bay Area Juniper Meetup
Automation in Network Lifecycle Management - Bay Area Juniper MeetupAutomation in Network Lifecycle Management - Bay Area Juniper Meetup
Automation in Network Lifecycle Management - Bay Area Juniper MeetupJorge Bonilla
 

What's hot (20)

OSMC 2021 | inspectIT Ocelot: Dynamic OpenTelemetry Instrumentation at Runtime
OSMC 2021 | inspectIT Ocelot: Dynamic OpenTelemetry Instrumentation at RuntimeOSMC 2021 | inspectIT Ocelot: Dynamic OpenTelemetry Instrumentation at Runtime
OSMC 2021 | inspectIT Ocelot: Dynamic OpenTelemetry Instrumentation at Runtime
 
Cloudify workshop at CCCEU 2014
Cloudify workshop at CCCEU 2014 Cloudify workshop at CCCEU 2014
Cloudify workshop at CCCEU 2014
 
OSMC 2021 || Open Source Application Performance Monitoring in the Enterprise
OSMC 2021 || Open Source Application Performance Monitoring in the EnterpriseOSMC 2021 || Open Source Application Performance Monitoring in the Enterprise
OSMC 2021 || Open Source Application Performance Monitoring in the Enterprise
 
Automation Evolution with Junos
Automation Evolution with JunosAutomation Evolution with Junos
Automation Evolution with Junos
 
OSMC 2021 | Current State of Icinga
OSMC 2021 | Current State of IcingaOSMC 2021 | Current State of Icinga
OSMC 2021 | Current State of Icinga
 
Elk ruminating on logs
Elk ruminating on logsElk ruminating on logs
Elk ruminating on logs
 
Developing with the Go client for Apache Kafka
Developing with the Go client for Apache KafkaDeveloping with the Go client for Apache Kafka
Developing with the Go client for Apache Kafka
 
Testing at Stream-Scale
Testing at Stream-ScaleTesting at Stream-Scale
Testing at Stream-Scale
 
Performance Comparison of Streaming Big Data Platforms
Performance Comparison of Streaming Big Data PlatformsPerformance Comparison of Streaming Big Data Platforms
Performance Comparison of Streaming Big Data Platforms
 
Getting Pulsar Spinning_Addison Higham
Getting Pulsar Spinning_Addison HighamGetting Pulsar Spinning_Addison Higham
Getting Pulsar Spinning_Addison Higham
 
Open source applied: Real-world uses
Open source applied: Real-world usesOpen source applied: Real-world uses
Open source applied: Real-world uses
 
How to build a Neutron Plugin (stadium edition)
How to build a Neutron Plugin (stadium edition)How to build a Neutron Plugin (stadium edition)
How to build a Neutron Plugin (stadium edition)
 
Open source: Top issues in the top enterprise packages
Open source: Top issues in the top enterprise packagesOpen source: Top issues in the top enterprise packages
Open source: Top issues in the top enterprise packages
 
Performance Testing using Real Browsers with JMeter & Webdriver
Performance Testing using Real Browsers with JMeter & WebdriverPerformance Testing using Real Browsers with JMeter & Webdriver
Performance Testing using Real Browsers with JMeter & Webdriver
 
Configuration Management Tools on NX-OS
Configuration Management Tools on NX-OSConfiguration Management Tools on NX-OS
Configuration Management Tools on NX-OS
 
Kafka aws
Kafka awsKafka aws
Kafka aws
 
Apache Spark the Hard Way: Challenges with Building an On-Prem Spark Analytic...
Apache Spark the Hard Way: Challenges with Building an On-Prem Spark Analytic...Apache Spark the Hard Way: Challenges with Building an On-Prem Spark Analytic...
Apache Spark the Hard Way: Challenges with Building an On-Prem Spark Analytic...
 
Interactive Analytics using Apache Spark
Interactive Analytics using Apache SparkInteractive Analytics using Apache Spark
Interactive Analytics using Apache Spark
 
Helium makes Zeppelin fly!
Helium makes Zeppelin fly!Helium makes Zeppelin fly!
Helium makes Zeppelin fly!
 
Automation in Network Lifecycle Management - Bay Area Juniper Meetup
Automation in Network Lifecycle Management - Bay Area Juniper MeetupAutomation in Network Lifecycle Management - Bay Area Juniper Meetup
Automation in Network Lifecycle Management - Bay Area Juniper Meetup
 

Viewers also liked

QCon London 2015 - Wrangling Data at the IOT Rodeo
QCon London 2015 - Wrangling Data at the IOT RodeoQCon London 2015 - Wrangling Data at the IOT Rodeo
QCon London 2015 - Wrangling Data at the IOT RodeoDamien Dallimore
 
SpringOne2GX 2014 Splunk Presentation
SpringOne2GX 2014 Splunk PresentationSpringOne2GX 2014 Splunk Presentation
SpringOne2GX 2014 Splunk PresentationDamien Dallimore
 
Splunking the JVM (Java Virtual Machine)
Splunking the JVM (Java Virtual Machine)Splunking the JVM (Java Virtual Machine)
Splunking the JVM (Java Virtual Machine)Damien Dallimore
 
Learn Dashing Widget in 90 minutes
Learn Dashing Widget in 90 minutesLearn Dashing Widget in 90 minutes
Learn Dashing Widget in 90 minutesLarry Cai
 
Java Aktuell Bernd Zuther Canary Releases mit der Very Awesome Microservices ...
Java Aktuell Bernd Zuther Canary Releases mit der Very Awesome Microservices ...Java Aktuell Bernd Zuther Canary Releases mit der Very Awesome Microservices ...
Java Aktuell Bernd Zuther Canary Releases mit der Very Awesome Microservices ...Bernd Zuther
 
Bau dein eigenes extreme feedback device
Bau dein eigenes extreme feedback deviceBau dein eigenes extreme feedback device
Bau dein eigenes extreme feedback deviceBernd Zuther
 
Splunk conf2014 - Dashboard Fun - Creating an Interactive Transaction Profiler
Splunk conf2014 - Dashboard Fun - Creating an Interactive Transaction ProfilerSplunk conf2014 - Dashboard Fun - Creating an Interactive Transaction Profiler
Splunk conf2014 - Dashboard Fun - Creating an Interactive Transaction ProfilerSplunk
 

Viewers also liked (9)

QCon London 2015 - Wrangling Data at the IOT Rodeo
QCon London 2015 - Wrangling Data at the IOT RodeoQCon London 2015 - Wrangling Data at the IOT Rodeo
QCon London 2015 - Wrangling Data at the IOT Rodeo
 
SpringOne2GX 2014 Splunk Presentation
SpringOne2GX 2014 Splunk PresentationSpringOne2GX 2014 Splunk Presentation
SpringOne2GX 2014 Splunk Presentation
 
Splunk Developer Platform
Splunk Developer PlatformSplunk Developer Platform
Splunk Developer Platform
 
Splunking the JVM (Java Virtual Machine)
Splunking the JVM (Java Virtual Machine)Splunking the JVM (Java Virtual Machine)
Splunking the JVM (Java Virtual Machine)
 
Learn Dashing Widget in 90 minutes
Learn Dashing Widget in 90 minutesLearn Dashing Widget in 90 minutes
Learn Dashing Widget in 90 minutes
 
Java Aktuell Bernd Zuther Canary Releases mit der Very Awesome Microservices ...
Java Aktuell Bernd Zuther Canary Releases mit der Very Awesome Microservices ...Java Aktuell Bernd Zuther Canary Releases mit der Very Awesome Microservices ...
Java Aktuell Bernd Zuther Canary Releases mit der Very Awesome Microservices ...
 
Bau dein eigenes extreme feedback device
Bau dein eigenes extreme feedback deviceBau dein eigenes extreme feedback device
Bau dein eigenes extreme feedback device
 
Splunk conf2014 - Dashboard Fun - Creating an Interactive Transaction Profiler
Splunk conf2014 - Dashboard Fun - Creating an Interactive Transaction ProfilerSplunk conf2014 - Dashboard Fun - Creating an Interactive Transaction Profiler
Splunk conf2014 - Dashboard Fun - Creating an Interactive Transaction Profiler
 
Using the Splunk Java SDK
Using the Splunk Java SDKUsing the Splunk Java SDK
Using the Splunk Java SDK
 

Similar to Splunk Conf 2014 - Splunking the Java Virtual Machine

Integrating Splunk into your Spring Applications
Integrating Splunk into your Spring ApplicationsIntegrating Splunk into your Spring Applications
Integrating Splunk into your Spring ApplicationsDamien Dallimore
 
Using Docker EE to Scale Operational Intelligence at Splunk
Using Docker EE to Scale Operational Intelligence at SplunkUsing Docker EE to Scale Operational Intelligence at Splunk
Using Docker EE to Scale Operational Intelligence at SplunkDocker, Inc.
 
Java Development on Bluemix
Java Development on BluemixJava Development on Bluemix
Java Development on BluemixRam Vennam
 
Heroku for-team-collaboration
Heroku for-team-collaborationHeroku for-team-collaboration
Heroku for-team-collaborationJohn Stevenson
 
Heroku for team collaboration
Heroku for team collaborationHeroku for team collaboration
Heroku for team collaborationJohn Stevenson
 
SpringOne 2016 in a nutshell
SpringOne 2016 in a nutshellSpringOne 2016 in a nutshell
SpringOne 2016 in a nutshellJeroen Resoort
 
Getting Started with Splunk Enterprise Hands-On Breakout Session
Getting Started with Splunk Enterprise Hands-On Breakout SessionGetting Started with Splunk Enterprise Hands-On Breakout Session
Getting Started with Splunk Enterprise Hands-On Breakout SessionSplunk
 
JavaOne 2015 CON7547 "Beyond the Coffee Cup: Leveraging Java Runtime Technolo...
JavaOne 2015 CON7547 "Beyond the Coffee Cup: Leveraging Java Runtime Technolo...JavaOne 2015 CON7547 "Beyond the Coffee Cup: Leveraging Java Runtime Technolo...
JavaOne 2015 CON7547 "Beyond the Coffee Cup: Leveraging Java Runtime Technolo...0xdaryl
 
J9: Under the hood of the next open source JVM
J9: Under the hood of the next open source JVMJ9: Under the hood of the next open source JVM
J9: Under the hood of the next open source JVMDanHeidinga
 
WebLogic 12c - OMF Canberra June 2014
WebLogic 12c - OMF Canberra June 2014WebLogic 12c - OMF Canberra June 2014
WebLogic 12c - OMF Canberra June 2014Joelith
 
Impact2014: Introduction to the IBM Java Tools
Impact2014: Introduction to the IBM Java ToolsImpact2014: Introduction to the IBM Java Tools
Impact2014: Introduction to the IBM Java ToolsChris Bailey
 
Creating Complete Test Environments in the Cloud: Skytap & Parasoft Webinar
Creating Complete Test Environments in the Cloud: Skytap & Parasoft WebinarCreating Complete Test Environments in the Cloud: Skytap & Parasoft Webinar
Creating Complete Test Environments in the Cloud: Skytap & Parasoft WebinarSkytap Cloud
 
JNUC 2017: Open Distribution Server
JNUC 2017: Open Distribution ServerJNUC 2017: Open Distribution Server
JNUC 2017: Open Distribution ServerBryson Tyrrell
 
Deep Learning for Java Developer - Getting Started
Deep Learning for Java Developer - Getting StartedDeep Learning for Java Developer - Getting Started
Deep Learning for Java Developer - Getting StartedSuyash Joshi
 
Cloud-powered Continuous Integration and Deployment architectures - Jinesh Varia
Cloud-powered Continuous Integration and Deployment architectures - Jinesh VariaCloud-powered Continuous Integration and Deployment architectures - Jinesh Varia
Cloud-powered Continuous Integration and Deployment architectures - Jinesh VariaAmazon Web Services
 
Liberate your Application Logging
Liberate your Application LoggingLiberate your Application Logging
Liberate your Application LoggingGlenn Block
 
SplunkLive! Amsterdam 2015 Breakout - Getting Started with Splunk
SplunkLive! Amsterdam 2015 Breakout - Getting Started with SplunkSplunkLive! Amsterdam 2015 Breakout - Getting Started with Splunk
SplunkLive! Amsterdam 2015 Breakout - Getting Started with SplunkSplunk
 
Was l iberty for java batch and jsr352
Was l iberty for java batch and jsr352Was l iberty for java batch and jsr352
Was l iberty for java batch and jsr352sflynn073
 

Similar to Splunk Conf 2014 - Splunking the Java Virtual Machine (20)

Integrating Splunk into your Spring Applications
Integrating Splunk into your Spring ApplicationsIntegrating Splunk into your Spring Applications
Integrating Splunk into your Spring Applications
 
Using Docker EE to Scale Operational Intelligence at Splunk
Using Docker EE to Scale Operational Intelligence at SplunkUsing Docker EE to Scale Operational Intelligence at Splunk
Using Docker EE to Scale Operational Intelligence at Splunk
 
Java Development on Bluemix
Java Development on BluemixJava Development on Bluemix
Java Development on Bluemix
 
Heroku for-team-collaboration
Heroku for-team-collaborationHeroku for-team-collaboration
Heroku for-team-collaboration
 
Heroku for team collaboration
Heroku for team collaborationHeroku for team collaboration
Heroku for team collaboration
 
SpringOne 2016 in a nutshell
SpringOne 2016 in a nutshellSpringOne 2016 in a nutshell
SpringOne 2016 in a nutshell
 
NZS-4409 - Enterprise Java Monitoring on zOS Discover, Alert, Optimize
NZS-4409 - Enterprise Java Monitoring on zOS Discover, Alert, OptimizeNZS-4409 - Enterprise Java Monitoring on zOS Discover, Alert, Optimize
NZS-4409 - Enterprise Java Monitoring on zOS Discover, Alert, Optimize
 
Getting Started with Splunk Enterprise Hands-On Breakout Session
Getting Started with Splunk Enterprise Hands-On Breakout SessionGetting Started with Splunk Enterprise Hands-On Breakout Session
Getting Started with Splunk Enterprise Hands-On Breakout Session
 
JavaOne 2015 CON7547 "Beyond the Coffee Cup: Leveraging Java Runtime Technolo...
JavaOne 2015 CON7547 "Beyond the Coffee Cup: Leveraging Java Runtime Technolo...JavaOne 2015 CON7547 "Beyond the Coffee Cup: Leveraging Java Runtime Technolo...
JavaOne 2015 CON7547 "Beyond the Coffee Cup: Leveraging Java Runtime Technolo...
 
Troubleshooting Tools In JDK
Troubleshooting Tools In JDKTroubleshooting Tools In JDK
Troubleshooting Tools In JDK
 
J9: Under the hood of the next open source JVM
J9: Under the hood of the next open source JVMJ9: Under the hood of the next open source JVM
J9: Under the hood of the next open source JVM
 
WebLogic 12c - OMF Canberra June 2014
WebLogic 12c - OMF Canberra June 2014WebLogic 12c - OMF Canberra June 2014
WebLogic 12c - OMF Canberra June 2014
 
Impact2014: Introduction to the IBM Java Tools
Impact2014: Introduction to the IBM Java ToolsImpact2014: Introduction to the IBM Java Tools
Impact2014: Introduction to the IBM Java Tools
 
Creating Complete Test Environments in the Cloud: Skytap & Parasoft Webinar
Creating Complete Test Environments in the Cloud: Skytap & Parasoft WebinarCreating Complete Test Environments in the Cloud: Skytap & Parasoft Webinar
Creating Complete Test Environments in the Cloud: Skytap & Parasoft Webinar
 
JNUC 2017: Open Distribution Server
JNUC 2017: Open Distribution ServerJNUC 2017: Open Distribution Server
JNUC 2017: Open Distribution Server
 
Deep Learning for Java Developer - Getting Started
Deep Learning for Java Developer - Getting StartedDeep Learning for Java Developer - Getting Started
Deep Learning for Java Developer - Getting Started
 
Cloud-powered Continuous Integration and Deployment architectures - Jinesh Varia
Cloud-powered Continuous Integration and Deployment architectures - Jinesh VariaCloud-powered Continuous Integration and Deployment architectures - Jinesh Varia
Cloud-powered Continuous Integration and Deployment architectures - Jinesh Varia
 
Liberate your Application Logging
Liberate your Application LoggingLiberate your Application Logging
Liberate your Application Logging
 
SplunkLive! Amsterdam 2015 Breakout - Getting Started with Splunk
SplunkLive! Amsterdam 2015 Breakout - Getting Started with SplunkSplunkLive! Amsterdam 2015 Breakout - Getting Started with Splunk
SplunkLive! Amsterdam 2015 Breakout - Getting Started with Splunk
 
Was l iberty for java batch and jsr352
Was l iberty for java batch and jsr352Was l iberty for java batch and jsr352
Was l iberty for java batch and jsr352
 

Recently uploaded

Novo Nordisk: When Knowledge Graphs meet LLMs
Novo Nordisk: When Knowledge Graphs meet LLMsNovo Nordisk: When Knowledge Graphs meet LLMs
Novo Nordisk: When Knowledge Graphs meet LLMsNeo4j
 
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Eraconfluent
 
Auto Affiliate AI Earns First Commission in 3 Hours..pdf
Auto Affiliate  AI Earns First Commission in 3 Hours..pdfAuto Affiliate  AI Earns First Commission in 3 Hours..pdf
Auto Affiliate AI Earns First Commission in 3 Hours..pdfSelfMade bd
 
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...Abortion Clinic
 
Weeding your micro service landscape.pdf
Weeding your micro service landscape.pdfWeeding your micro service landscape.pdf
Weeding your micro service landscape.pdftimtebeek1
 
OpenChain Webinar: AboutCode and Beyond - End-to-End SCA
OpenChain Webinar: AboutCode and Beyond - End-to-End SCAOpenChain Webinar: AboutCode and Beyond - End-to-End SCA
OpenChain Webinar: AboutCode and Beyond - End-to-End SCAShane Coughlan
 
Encryption Recap: A Refresher on Key Concepts
Encryption Recap: A Refresher on Key ConceptsEncryption Recap: A Refresher on Key Concepts
Encryption Recap: A Refresher on Key Conceptsthomashtkim
 
A Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdfA Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdfICS
 
Effective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeConEffective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeConNatan Silnitsky
 
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdfAzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdfryanfarris8
 
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...Neo4j
 
Software Engineering - Introduction + Process Models + Requirements Engineering
Software Engineering - Introduction + Process Models + Requirements EngineeringSoftware Engineering - Introduction + Process Models + Requirements Engineering
Software Engineering - Introduction + Process Models + Requirements EngineeringPrakhyath Rai
 
Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024Henry Schreiner
 
Prompt Engineering - an Art, a Science, or your next Job Title?
Prompt Engineering - an Art, a Science, or your next Job Title?Prompt Engineering - an Art, a Science, or your next Job Title?
Prompt Engineering - an Art, a Science, or your next Job Title?Maxim Salnikov
 

Recently uploaded (20)

Novo Nordisk: When Knowledge Graphs meet LLMs
Novo Nordisk: When Knowledge Graphs meet LLMsNovo Nordisk: When Knowledge Graphs meet LLMs
Novo Nordisk: When Knowledge Graphs meet LLMs
 
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Era
 
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
 
Abortion Pill Prices Jozini ](+27832195400*)[ 🏥 Women's Abortion Clinic in Jo...
Abortion Pill Prices Jozini ](+27832195400*)[ 🏥 Women's Abortion Clinic in Jo...Abortion Pill Prices Jozini ](+27832195400*)[ 🏥 Women's Abortion Clinic in Jo...
Abortion Pill Prices Jozini ](+27832195400*)[ 🏥 Women's Abortion Clinic in Jo...
 
Auto Affiliate AI Earns First Commission in 3 Hours..pdf
Auto Affiliate  AI Earns First Commission in 3 Hours..pdfAuto Affiliate  AI Earns First Commission in 3 Hours..pdf
Auto Affiliate AI Earns First Commission in 3 Hours..pdf
 
Abortion Clinic In Stanger ](+27832195400*)[ 🏥 Safe Abortion Pills In Stanger...
Abortion Clinic In Stanger ](+27832195400*)[ 🏥 Safe Abortion Pills In Stanger...Abortion Clinic In Stanger ](+27832195400*)[ 🏥 Safe Abortion Pills In Stanger...
Abortion Clinic In Stanger ](+27832195400*)[ 🏥 Safe Abortion Pills In Stanger...
 
Abortion Pill Prices Turfloop ](+27832195400*)[ 🏥 Women's Abortion Clinic in ...
Abortion Pill Prices Turfloop ](+27832195400*)[ 🏥 Women's Abortion Clinic in ...Abortion Pill Prices Turfloop ](+27832195400*)[ 🏥 Women's Abortion Clinic in ...
Abortion Pill Prices Turfloop ](+27832195400*)[ 🏥 Women's Abortion Clinic in ...
 
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
 
Weeding your micro service landscape.pdf
Weeding your micro service landscape.pdfWeeding your micro service landscape.pdf
Weeding your micro service landscape.pdf
 
OpenChain Webinar: AboutCode and Beyond - End-to-End SCA
OpenChain Webinar: AboutCode and Beyond - End-to-End SCAOpenChain Webinar: AboutCode and Beyond - End-to-End SCA
OpenChain Webinar: AboutCode and Beyond - End-to-End SCA
 
Encryption Recap: A Refresher on Key Concepts
Encryption Recap: A Refresher on Key ConceptsEncryption Recap: A Refresher on Key Concepts
Encryption Recap: A Refresher on Key Concepts
 
A Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdfA Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdf
 
Effective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeConEffective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeCon
 
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdfAzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
 
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
 
Software Engineering - Introduction + Process Models + Requirements Engineering
Software Engineering - Introduction + Process Models + Requirements EngineeringSoftware Engineering - Introduction + Process Models + Requirements Engineering
Software Engineering - Introduction + Process Models + Requirements Engineering
 
Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024
 
Abortion Pill Prices Mthatha (@](+27832195400*)[ 🏥 Women's Abortion Clinic In...
Abortion Pill Prices Mthatha (@](+27832195400*)[ 🏥 Women's Abortion Clinic In...Abortion Pill Prices Mthatha (@](+27832195400*)[ 🏥 Women's Abortion Clinic In...
Abortion Pill Prices Mthatha (@](+27832195400*)[ 🏥 Women's Abortion Clinic In...
 
Prompt Engineering - an Art, a Science, or your next Job Title?
Prompt Engineering - an Art, a Science, or your next Job Title?Prompt Engineering - an Art, a Science, or your next Job Title?
Prompt Engineering - an Art, a Science, or your next Job Title?
 
Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...
Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...
Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...
 

Splunk Conf 2014 - Splunking the Java Virtual Machine

  • 1. Copyright © 2014 Splunk Inc. Splunking the JVM Damien Dallimore Dev Evangelist , CSO Office @ Splunk
  • 2. Disclaimer During the course of this presentation, we may make forward looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in the this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release. 2
  • 3. 3 From Middle Earth Make Splunk Apps & Add-ons JVM background
  • 4. 4
  • 6. What is this JVM thing ?
  • 7. JVM = Java Virtual Machine First appeared in the early 90’s Now the dominant runtime for enterprise applications Application Servers Enterprise Service Buses Databases NoSQL Distributed Big Data Web Servers Directory Servers Search Engines Build Systems Gaming Platforms Trading Systems Reservation Systems Core Banking Messaging Infrastructure Proprietary Systems 7
  • 8. 8
  • 9. Many JVM Variants Oracle Hotspot OpenJDK Oracle JRockit 9 IBM J9 Azul Zing More or less the same. OpenJDK is reference impl for Java 7/8 Getting merged into Hotspot Commercial alternatives
  • 10. JVMs are here to stay It’s not just about the “J” Big Data frameworks You should care about getting insights into your JVM data 10
  • 12. Data Sources Logs Developers JMX (Java Management Extensions) Instrumentation Agents Operating System Distributed Communications 12
  • 14. Standard Log Files Oct 21, 2013 4:42:15 PM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 1153 ms Oct 21, 2013 4:42:15 PM org.apache.catalina.core.StandardService startInternal INFO: Starting service Catalina Application logs that are part of the product Developer logs for any code that was deployed 14 Written to local disk or network storage
  • 15. Garbage Collector Logs 54.736: [Full GC 54.737: [Tenured: 172798K->18092K(174784K), 2.3792658 secs] 257598K->18092K(259584K), [Perm : 20476K->20476K(20480K)], 2.4715398 secs] [Times: user=0.56 sys=0.05, real=0.07 secs] Generated by way of arguments passed to the JVM at startup 15 -verbose:gc -Xloggc:/home/damien/jvm_logs/gc.log -XX:+PrintGC -XX:+PrintGCTimeStamps -XX:+PrintGCDetails
  • 16. SplunkJavaLogging Sometimes you can’t write to file or deploy a UF Appenders for Java Util Logging , Log4J , Logback Simply add a logging appender to your logging configuration file 16
  • 19. Best practice semantic format 2012-08-07 15:54:06:644+1200 name="Failed Login" event_id=”12345" app="myapp" user="jane” 19
  • 21. Easier to work with in Splunk 21
  • 22. Splunk SDK for Java Use the SDK from any JVM Language , Java / Groovy / Scala etc…. Send log events via REST , UDP or TCP directly to Splunk from your code Spring Integration Adaptors available on Github 22 SDK available from dev.splunk.com
  • 25. JMX
  • 26. JMX = Java Management Extensions Monitor JVM via MBean attributes , operations and notifications JVM MBeans Vendor MBeans Custom Coded MBeans 26
  • 27. Browse MBeans with JConsole 27
  • 28. Getting this data into Splunk Runs on all supported Splunk platforms Works with all main JVM variants 100% Free and Open Source 28
  • 30. Many Connectivity Options Let’s look at this Splunk App in action 30
  • 31. 31 This is great for monitoring a single JVM……
  • 32. But what if we have dozens , hundreds , even thousands ? 32
  • 33. Multiple jmxserver elements per configuration file 33
  • 34. Multiple configuration files per Splunk Instance 34
  • 35. But stacking vertically will only get you so far 35
  • 36. Go horizontal to achieve real scale 36 Indexer Cluster UFs with JMX Mod Input Monitored JVMs
  • 37. In my experience , the data sources shown so far will be thorough enough for most of your JVM monitoring use cases. But what if you want to go deeper ? 37
  • 39. SplunkJavaAgent Dynamic Byte Code Injection Agent for JVMs No need to change any code to get APM level insights This is all you pass to the JVM at startup : -javaagent:splunkagent.jar 39
  • 40. What raw metrics can you get ? 40 Bytecode Injection • class loading • method execution • method timings • method call stack • caught & uncaught exceptions JMX • running in local in-memory mode Binary HPROF Memory dumps decoded in text • no more static hprof dumps and then loading into another Mem Analysis tool
  • 42. 42
  • 43. Raw events streamed into Splunk 43
  • 47. The JVM executes as a process Let’s see what the host Operating System can tell us : top External programs that provide additional JVM insights : jstat Index this data in Splunk and correlate 47
  • 49. 49
  • 51. 51
  • 53. Valuable data flows in & out of JVMs 53 Messaging Capture packets off the wire
  • 54. Build a Splunk App !
  • 55. Opportunity Reuse some of the data collection approaches in this presentation and build out a specific JVM app 55 Publish it on apps.splunk.com Make some money Generate publicity for yourself / your company Generate sales leads Sell more Splunk licenses if you are also a partner Do it because you love community collaboration Become my beer buddy for life
  • 56. Some JVM App Ideas Splunk for JBoss Splunk for Tomcat Splunk for Solr Splunk for Cassandra Splunk for “Apache Project” I am here to help you 56
  • 57. THANK YOU @damiendallimore ddallimore@splunk.com

Editor's Notes

  1. Legal disclaimer
  2. From Auckland Dev evang , ex customer 5th Conf Make Apps , Cut code Through enterprise Java background in many verticals , enterprise and non enterprise Java and other JVM langs
  3. Everything 100% open source use , reuse , whatever. Collaborate Community answers.splunk.com for support is best
  4. JVM is a runtime for code that gets compiled to bytecode and executed. Not natively. But can JIT. So performance is good now. Circa 1991, Dr. James Gosling at Sun started developing a technology for next generation smart devices/appliances “Green” became “Oak” which became “Java” Java 1.0 first appeared in January 1996. The JVM is a virtual machine that runs programs that are compiled into Java bytecode Available for many hardware and software platforms 17 years later , the JVM has evolved from a consumer device technology, to a browser oriented technology with the explosion of the web , to now becoming deeply rooted in the enterprise software landscape on the server side and in the cloud Overcame some limitatons on C/C++ Language changes , generics , annotations , lambda’s , streams , default methods GC , JIT , bytecode , write once run anywhere
  5. Ask crowd about JVM experience : Devs Operations Have no idea but might after this preso
  6. Jrockit and hotspot code being integrated Openjdk closely linked to hotspot , Oracle JDK has a few other proprietary add-ons above. Open JDK is the Java reference impl Zing is commercial
  7. Groovy , Scala , Clojure , JS , the JVM is evolving with the changing landscape of enterprise software From app servers in the 00’s to Big Data frameworks today Hadoop , Cassandra, 1000’s of nodes The JVM footprint cross cuts the data centre and represents a massive source of valuable machine data Large scale Application/Web Server clusters Hadoop & Cassandra Node topologies in the 100’s and in some cases 1000’s Segway into JVM Data
  8. Monitor with a UF = best practice
  9. JVM generated logs by way of options
  10. Try to use a UF if at all possible Utility classes for formatting log events Configurable in memory buffer to handle network outages
  11. TCP and REST Appenders
  12. SplunkLogEvent will log in best practive semantic format
  13. index=main sourcetype=log4j2 | makemv delim="," stacktrace_elements | table _time throwable_message throwable_class stacktrace_elements
  14. What if you can’t use logging appenders etc… Well , then you have to get down and dirty with your code !
  15. Majority of useful data for monitoring will come from this layer Open and easily extensible Developers can simply create new MBeans Vendor products(JBoss, Cassandra, Hadoop etc..) ship with thorough MBean coverage, not MIBs
  16. JConsole to Browse Mbeans Enable your target JVM’s remote JMX interface , test connectivity with JConsole
  17. Splunkcowboy.com for demo
  18. As many config files as you want Can make output CIM compliant
  19. Also MX4j connectivity
  20. Run Splunk UF locally with target JVM.Can connect use remote JMX interface or direct process attachment. Each tier scales out horizontally. Can overcome firewall issues that are sometimes inherent with Java RMI Deploy Splunk for JMX components and configurations with Splunk Deployment Server, Puppet or Chef. More than just jvm as bottleneck , also have stdin/out and Splunk indexing pipeline
  21. Appdynamics have an integration
  22. Hprof : Binary JVM dumps that allow for deeper JVM resource inspection Typical use case is diagnosing memory issues after JVM crashes with java.lang.OutOfMemoryError
  23. sourcetype=splunk_java_agent name=method* | transaction appName,appID,threadID,className,methodName startswith=method_entered endswith=method_exited | stats min(duration) as "Min Execution Time" max(duration) as "Max Execution Time" avg(duration) as "Avg Execution Time" count as "Total Calls" sum(duration) as "Cumulative Execution Time" by className,methodName
  24. Correlate this OS data across your JVM and Application events ie: your JVM may have hung because of CPU starvation caused by some other process thrashing Could even go lower if running virtual and get the VM/Hypervisor data
  25. Created a wrapper script to dynamically get the PID of the Java process
  26. Custom handlers for pre-processing and reformatting if you want
  27. Not going into details here , there are other sessions you can attend.
  28. ie: reuse the JMX Modular input