Splunk for JMX
Upcoming SlideShare
Loading in...5

Splunk for JMX



Presentation section from Splunk Live content

Presentation section from Splunk Live content



Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

Splunk for JMX Splunk for JMX Presentation Transcript

  • Splunk for JMXDamien DallimoreDeveloper Evangelist
  • Copyright©2013,SplunkInc.Splunk for JMX2• ConnecttoanylocalorremoteJVMsJMXserver,Hotspot/JRockit/IBMJ9/OpenJDK/AzulZing• QueryanyMBeanrunningonthatserver• ExtractanyMBeanattributes(simple,compositeortabular)• InvokeMBeanoperations• Writeattributesandoperationresultsoutinadefaultkey/valueformat,orpluginyourowncustomformat,forSPLUNKindexingandsearching• TransporteventsoverSTDOUT(default),TCP,Syslog,SplunkRESTendpointordirecttofile.• DeclareclustersofJVMsforlargerscaleJVMdeployments• Runson*NixandWindows• OutoftheboxdashboardsforcommonJVMMBeans• FreelyavailablefromSplunkBase,allsourcecodeisonGitHub
  • Copyright©2013,SplunkInc.Connectivity Options3• Remote JMX interface– rmi (JSR160StandardImplementationandMX4JsJSR160Implementation)– iiop(JSR160StandardImplementationandMX4JsJSR160Implementation)• Direct Process attachment– ConnectdirectlytoalocallyrunningJVMprocess• MX4J HTTP connectors (requires MX4J in the target JVM also)– soap,soap+sssl– hessian,hessian+ssl– burlap,burlap+ssl
  • Copyright©2013,SplunkInc.Setup and Configuration4• The main goalof the app was to make it as simple and intuitive as possible to connect toyour JVMs and start Splunking JMX data• Enableyour targetJVM’s remoteJMX interface, test connectivitywith JConsole• InstallSplunk for JMX• ExtractSplunkforJMXtarballtoSPLUNK_HOME/etc/apps• RestartSplunk• Atthesetupscreen,choosea scriptedinputforyourplatform(Nix/Windows)• Setup your JMX configurationfile• Thedefaultconfig.xmlfileispreconfiguredforcommonJVMMBeans• BrowseyourJVM(usingJConsole)forotherMBeansthatyouwishtopollandconfigurethese• Youcanhaveasmanyconfigfilesasyourequire,andyoumightsettheseuptofireoffatdifferentscheduledfrequencies
  • Copyright©2013,SplunkInc.Configuration Examples - Simple5
  • Copyright©2013,SplunkInc.Configuration Examples - Wildcards6MBean Object name format “domain:key=value,key2=value2”Around 25KBytes per dump on Hotspot JVMsDon’t know the domain or properties name or have many Mbeanswith the same attributes ?* and ? wildcards are supported in the Mbean name
  • Copyright©2013,SplunkInc.Configuration Examples - Clusters7• Define clusters of JVM’s that share thesame MBean definitions• Note , in these examples, for brevity I amusing “dumpAllAttributes” , but inproduction you’d want to pick andchoose specific MBean attributes youare interested in, and perhaps splitdefinitions over multiple files run atvarying frequencies
  • Copyright©2013,SplunkInc.Configuration Examples - Operations8• Invoke JMX operations that return a value or simplyperform some action on the target JVM• Operation definitions can take parametersUse Case 1 : your developers might code a JMXoperation that returns a CSV or JSON formattedsnapshot of some metrics for SplunkingUse Case 2 : dynamically trigger HPROF dumps.The “com.sun.management:type=HotSpotDiagnostic”Mbean exposes a “dumpHeap” operation
  • Copyright©2013,SplunkInc.Configuration Examples - Connecting9• IP Address with credentials• Hostname• Static Process ID• Process ID lookup from file• Process ID lookup from command output• Raw JMX Service URL• MX4J HTTP Connector
  • Copyright©2013,SplunkInc.Custom Formatters/Transports10• The Splunk for JMX configuration is user extensible• You can code and configure your own Formatters and TransportsFormatters• Takes the raw MBean polled output and formats it for Splunking• A Java implementation of the "com.dtdsoftware.splunk.formatter.Formatter" interface• If the optional formatter declaration is omitted, then the default formatter will be usedTransports• Takes the formatted output and transports it to a destination• A Java implementation of the "com.dtdsoftware.splunk.transport.Transport" interface• If the optional transport declaration is omitted, then the default transport(STD out) will be used
  • Copyright©2013,SplunkInc.Formatter Examples11
  • Copyright©2013,SplunkInc.Transport Examples12
  • Copyright©2013,SplunkInc.Deployment Architectures 113• Simplest scenario• Monolithic Splunk installation• Splunk for JMX polling 1 or more remote/local JVMs viathe remote JMX interface• There is support for many target JVM’s in theconfiguration schema but to really scale out, you needa more advanced Splunk architecture
  • Copyright©2013,SplunkInc.Deployment Architectures 214Splunk UF running locallywith target JVMSplunk IndexerClusterSplunk Search HeadPoolLoad Balancer• Run Splunk UF locally with targetJVM.Can connect use remote JMXinterface or direct process attachment.• Each tier scales out horizontally.• Can overcome firewall issues that aresometimes inherent with Java RMI• Deploy Splunk for JMX components andconfigurations with Splunk DeploymentServer, Puppet or Chef.
  • Demo
  • Copyright©2013,SplunkInc.Contact me16Email : ddallimore@splunk.comTwitter : @damiendallimoreSkype : damien.dallimoreGithub : damiendallimoreSplunkbase : damiendSlideshare : http://www.slideshare.net/damiendallimoreBlogs : http://blogs.splunk.com/devWeb : http://dev.splunk.com