Successfully reported this slideshow.
Presentation section from Splunk Live content
Splunking the JVMDamien DallimoreDeveloper Evangelist
Copyright©2013,SplunkInc.What is this JVM thing ?2• Circa 1991, Dr. James Gosling at Sun started developing a technologyfor next generation smart devices/appliances• “Green” became “Oak” which became “Java”• Java 1.0 first appeared in January 1996.• The JVM is a virtual machine that runs programs that are compiledinto Java bytecode• Available for many hardware and software platforms• 17 years later , the JVM has evolved from a consumer devicetechnology,to a browser oriented technology with the explosion ofthe web , to now becoming deeply rooted in the enterprisesoftwarelandscape on the server side and in the cloud
Copyright©2013,SplunkInc.17 years later3• Oracle took ownership of Java from Sun inJanuary 2010• The Java Community Process(JCP) is theforum where members developspecifications for Java technology• Java Specification Requests(JSR) getsubmitted for new features, are reviewed andthen voted on by the JCP Executivecommittee.• Editions• Embedded Java, Java ME , Java SE , Java EE• Current Version is Java 7 (Dolphin)• Java 8 scheduled for 2013Application Servers Enterprise Service Buses DatabasesNoSQL Distributed Big Data Web ServersDirectory Servers Search Engines Build SystemsGaming Platforms Trading Systems Reservation SystemsCore Banking Messaging Infrastructure Proprietary Systems
Copyright©2013,SplunkInc.JVM Variants4• Oracle Hotspot (formerly SUN)– theprimaryreferenceJVMimplementation• Oracle JRockit (formerly BEA)– freesinceMay2011– codebasecurrentlybeingmergedwithHotspot,ETA~JDK 8• Open JDK– SUN opensourcedHotspotand the Java classlibraryin 2006– SlightdifferenceswithOracleJava still– OpenJDKis the official JavaSE7 ReferenceImplementation• J9– IBM’sJVMforAIX,Linux,MVS, OS/400, PocketPC, z/OS• Azul Systems Zing– basedonHotSpot– supportsmemoryheapsup to 512 GB withoutGCpausesand is ableto growand shrinkthe heapbasedonload
Copyright©2013,SplunkInc.The JVM has a healthy future5• Hotspot/JRockitcodemergecreating abestofbreedJVM,OracletocontributethistoOpenJDK• OpenJDKisthriving,OraclearecontributingandbeinggoodstewardsofJava(despiteinitialskepticism)• Proliferationof alternativeJVMlanguagesthatcanallco-habitateintheJVMandnewfeaturesinJava8tofurtherenhancethismultilanguageplatform– Scala– Groovy– Clojure• TheJVMisevolvingorganicallywiththeshiftingtidesofEnterprisesoftware,itisn’taboutthe“J”anymore.• FromtheclusteredApplicationServerdominationofthe00’swenowseeanexplosionofBigDataproductsrunninginmassivelydistributedenvironmentsoncommodityhardwareorinthecloud– ApacheHadoopfamily(MapReduce,Hive,Hbase,Cassandra,HDFS)
Copyright©2013,SplunkInc.What is running in JVMs ?6
Copyright©2013,SplunkInc.JVM “Fanboi”7Dr. GoslingFanboiSpeaking of Java as a language as opposedto the JVM platform, James Gosling, theFather of Java, said "Most people talkabout Java the language, and this maysound odd coming from me, but I couldhardly care less."He went on to explain, "What I really careabout is the Java Virtual Machine as aconcept, because that is the thing that tiesit all together."
Copyright©2013,SplunkInc.JVM Machine Data8• The JVM footprint cross cuts the data centre and represents a massive source of valuable machine data• Large scale Application/Web Server clusters• Hadoop & Cassandra Node topologies in the 10’000s !!!Custom DevelopedCodeWAR fileApplication CodeTomcatJVMHotspotOperating SystemLinuxJMX, SNMP, HPROF,GC Logs, Custom Agents(AppDynamics/SplunkJavaAgent)JMX, Application LogsJMX, Developer Logs, Splunk Java SDK, SplunkJavaLoggingJVM process OS resource metricsCORRELATE
Copyright©2013,SplunkInc.Application & Developer Logs9• Application logs• default logs that are part of the product• Developer logs• any custom code created and deployedto the application that has it’s ownlogging• Written to local disk or a mounted networkvolume• Monitor with a Splunk UFSplunk IndexerSplunk Universal ForwarderMonitor Log Files/ DirectorysDeveloped CodeApplicationJVMOS
Copyright©2013,SplunkInc.Splunk Java SDK / SplunkJavaLogging10Splunk IndexerDeveloped CodeApplicationJVMOSHTTP$REST$/$TCP$/$UDP • Alternative to writing to log file orneeding to deploy a Splunk UniversalForwarder• Use the Splunk Java SDK to input eventsdirectly to Splunk via HTTP Rest.• Use SplunkJavaLogging to input eventsdirectly to Splunk using custom loggingappenders.
Copyright©2013,SplunkInc.JVM Process OS Metrics11• By JVM Process ID : Process State, Memory, CPU,Disk Usage, Disk I/O, Network I/O, FileDescriptor Usage.• Some OS metrics also exposed via JMX• Splunk for Unix and Linux• Splunk for Windows• Correlate this OS data across your JVM andApplication events ie: your JVM may have hungbecause of CPU starvation caused by some otherprocess thrashingSplunk IndexerSplunk for Unix or LinuxMonitor Log Files &DirectorysDeveloped CodeApplicationJVMOSPoll output fromcommands
Copyright©2013,SplunkInc.Garbage Collection logs12Splunk IndexerSplunk Universal ForwarderMonitor GC Log FilesDeveloped CodeApplicationJVMOS• Extended Hotspot JVM options-verbose:gc-Xloggc:/home/damien/jvm_logs/gc.log-XX:+PrintGC-XX:+PrintGCTimeStamps-XX:+PrintGCDetails• The log is written to at Garbage Collection time• Useful for tracing full GC cycles• Need to perform field extractions in Splunk• Many GC metrics also available via JMX54.736: [Full GC 54.737:[Tenured: 172798K->18092K(174784K), 2.3792658 secs] 257598K->18092K(259584K),[Perm : 20476K->20476K(20480K)], 2.4715398 secs] [Times: user=0.56 sys=0.05, real=0.07 secs]
Copyright©2013,SplunkInc.Custom Instrumentation Agents (Advanced)13Splunk IndexerSplunk UniversalForwarderMonitor Agent Log FilesDeveloped CodeApplicationJVMOSREST/TCP/UDP• JVM BCI (byte code instrumentation)• Write custom agents that get injected intothe running JVM• Dynamically inspect the state ofapplications running in the JVM• Profiling, debugging, monitoring,thread/memory analysis• As you write the agent code , the dataoutput can be file based or over thenetwork• Check out my SplunkJavaAgent on github• Also AppDynamics have some pretty coolkung fu in this area, we integrate !
Copyright©2013,SplunkInc.HPROF Profiling Dumps14Splunk IndexerSplunk UniversalForwarderBinary HPROF dump ﬁleDeveloped CodeApplicationJVMOSMonitor and decode intotextual key=value pairs• Binary JVM dumps that allow for deeper JVM resourceinspection• Typical use case is diagnosing memory issues after JVMcrashes with java.lang.OutOfMemoryError• Binary file is usually batch loaded into a third partymemory analysis tool like Eclipse MAT• SplunkJavaAgent can dynamically dump and decodehprof output and send to Splunk• Awesome source of information for dev/testWarning : heap dumping is an expensive operation as a full GC gets performed
Copyright©2013,SplunkInc.SNMP15• The JVM SNMP Agent provides a single MIB that exposes theJVM’s Management and Monitoring APIhttp://docs.oracle.com/javase/1.5.0/docs/guide/management/JVM-MANAGEMENT-MIB.mib• Setup the JVM (just the basic settings shown)Open a UDP Port : -Dcom.sun.management.snmp.port=9004Configure the ACL : $JAVA_HOME/jre/lib/management/snmp.acl• Traps can be caught locally to file and monitored• Splunk SNMP Modular Input can poll the JVM SNMP Objects(coming soon to a theatre near you)Splunk IndexerSplunk UniversalForwarderDeveloped CodeApplicationJVMOSSNMP%Objects%PolledJVM MIBsnmptrapd UDP:162SNMP%Traps%wri6en%to%file
Copyright©2013,SplunkInc.JMX (Java Management Extensions)16Splunk IndexerDeveloped CodeApplicationJVMOSSplunk UniversalForwarderJMX• Manage and Monitor the JVM and Application viaexposed MBeans• JVM MBeans (java.lang domain)• Vendor MBeans (most vendors ship their products withextensive MBean coverage)• Custom Coded MBeans (whatever your devs wish tocode)• MBeans expose attributes, operations andnotifications to give you a powerfully dynamicinsight into the runtime state of the JVM and yourapplication.• Add Splunk to the mix for historical and realtimeoperational visibility, pro-active issue detectionetc..• Splunk for JMX app on SplunkBase
Copyright©2013,SplunkInc.JMX vs SNMP17JMX• Open and easily extensible• Developers can simply create new MBeans• Vendor products(JBoss, Cassandra, Hadoop etc..) ship with thorough MBean coverage, not MIBsSNMP• The built-in SNMP agent of the JVM is not extensible.• You will not be able to use it in order to expose your own custom MIB• If you do want to expose your own MIB, you’d have to create a custom agent
Copyright©2013,SplunkInc.Putting it all together, JVM Splunking Nirvana18Splunk Indexer ClusterDeveloped CodeApplicationJVMOSJMXHPROFOS*Metrics/LogsSplunk ForwarderLogsJMXREST/TCP/UDPAuto Load BalancedJMXLogsDistributed Search
Copyright©2013,SplunkInc.Contact me19Email : email@example.comTwitter : @damiendallimoreSkype : damien.dallimoreGithub : damiendallimoreSplunkbase : damiendSlideshare : http://www.slideshare.net/damiendallimoreBlogs : http://blogs.splunk.com/devWeb : http://dev.splunk.com