• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Splunk Modular Inputs / JMS Messaging Module Input
 

Splunk Modular Inputs / JMS Messaging Module Input

on

  • 1,481 views

Presentation section from Splunk Live content

Presentation section from Splunk Live content

Statistics

Views

Total Views
1,481
Views on SlideShare
1,481
Embed Views
0

Actions

Likes
1
Downloads
24
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Splunk Modular Inputs / JMS Messaging Module Input Splunk Modular Inputs / JMS Messaging Module Input Presentation Transcript

    • Splunk Modular InputsDamien DallimoreDeveloper Evangelist
    • Copyright©2013,SplunkInc.Modular Inputs2• Extend the Splunk framework to define a custom input capability, just like the standard inputs youare familiar with (TCP/UDP/File etc…)• Splunk treats your custom input definitions as if they were part of Splunks native inputs, totallyintegrated first class citizen objects in Splunk• Users interactively create and update your custom inputs using Splunk manager, just as they do fornative inputs. When deploying without a UI , you push out the inputs.conf file.• All the properties are fully manageable via the REST API• Version 5.0 +
    • Copyright©2013,SplunkInc.What about scripted inputs ?3• Very loosely coupled to Splunk• No standard configuration/schema framework• No standard validation framework• No standard lifecycle management• Need to use “hacks” to make them running persistently• Not really integrated with the REST API• Logging not integrating with standard Splunk logsBUT• Their simplicity and loose coupling make them very rapid to develop• Choose the right tool for the job
    • Copyright©2013,SplunkInc.Diagram of Mod Input lifecycle4SplunkDInit / Request SchemeMod InputReturn SchemeExternal ValidateConfirm ValidationExecuteXMLXMLXMLStream ResultsText /XMLValidationCode &Error Msg$SPLUNK_HOME/var/log/splunk/splunkd.loglogging
    • Copyright©2013,SplunkInc.Scheme XML5
    • Copyright©2013,SplunkInc.Input XML6$SPLUNK_HOME/bin/splunk cmd splunkd print-modinput-config myscheme mystanza
    • Copyright©2013,SplunkInc.Manage Mod Inputs via REST API7
    • Copyright©2013,SplunkInc.A few other technical features8• Validation• External mode or via REST create/edit• Run Mode• single or multiple instance• Checkpoint directory• So your modular input can maintain state• Streaming Mode• Text or XML• XML streaming has more syntactic sugar for meta data, event breaking• Architecture specific scripts• Splunk auto magically chooses the correct runtime script.
    • Copyright©2013,SplunkInc.How are Mod Inputs going to help us9• We need to make it easy as possible to develop modular inputs , frameworks and tools• Sometimes the greatest battle is just getting the data in , modular inputs are a great tool in ourarmory.• Bundle Modular Inputs in with the core product (DB, JMX, SNMP, JMS etc…)• We need to make it easy to search for, install and configure these “data connectors”
    • Copyright©2013,SplunkInc.Developing10• My preference is to use Python, however any language can be used.• http://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/ModInputsIntro• There is a certain amount of “plumbing” to put in place , so I like utilitys that take care of this foryou, so you can just focus on the business logic.• I created utilitys to allow developers to rapidly create Modular Inputs in Java and Python• https://github.com/damiendallimore/SplunkModularInputsJavaFramework• https://github.com/damiendallimore/SplunkModularInputsPythonFramework• HelloWorld examples to get you started• Java -> JMS Messaging Modular Input , on Splunkbase• Python -> SNMP Modular Input , soon to be released
    • Copyright©2013,SplunkInc.Mod Inputs on Splunkbase11
    • Copyright©2013,SplunkInc.Messaging12• Message Oriented Middleware (MOM) infrastructures facilitate the sending/receiving ofmessages between distributed systems• Topics (publish/subscribe) and Queues (point to point)• The glue that stitches heterogeneous enterprise computing environments together• Represents a massive source of machine data that can be fed into Splunk to derive operationalvisibility into your messaging environment and the various systems and applications that arecommunicating via MOM
    • Copyright©2013,SplunkInc.Building a Splunk Messaging Solution13• There has been considerable demand for functionality in Splunk to index messagesfrom queues/topics• Ad hoc, proprietary, roll your own solutions were the only way• I wanted to develop an integrated mechanism to allow Splunk users to connect totheir MOM and index their messages• Modular Inputs provided the perfect platform to build a messaging solution
    • Copyright©2013,SplunkInc.JMS Messaging Modular Input14• JMS is simply a messaging interface that abstracts your underlying MOM providerimplementation• Most MOM vendors support JMS• So this allowed for creating 1 single modular input that can index messages from :• MQ Series / Websphere MQ• Tibco EMS• ActiveMQ• HornetQ• RabbitMQ• SonicMQ• JBoss Messaging• Weblogic JMS• Native JMS• StormMQ• MSMQ (with a bit of stuffing around)• Etc…• Simple to install : download from Splunkbase, drop in your apps directory, restart Splunk
    • Copyright©2013,SplunkInc.Key Features15• Known to work with all aforementioned Messaging platforms• Should work against any MOM platform with a JMS provider• Runs on all supported Splunk platforms• Consume messages from Topics and Queues• Browse Queues (if you don’t want to consume the messages) and just Splunkqueue stats• Messages header, properties and body indexed in Splunk in simple key/valuepairs• Can plug in your own message handler if you require customized processing ofthe message body• Authentication and SSL support• Scales horizontally if you require large volume message consumption
    • Copyright©2013,SplunkInc.JMS input fully integrated into Splunk16
    • Copyright©2013,SplunkInc.Add a new queue/topic input17
    • Copyright©2013,SplunkInc.Configure the properties to connect18
    • Copyright©2013,SplunkInc.Get instant operational visibility19
    • DemosJMS (ActiveMQ , Websphere MQ)SNMP
    • Copyright©2013,SplunkInc.Contact me21Email : ddallimore@splunk.comTwitter : @damiendallimoreSkype : damien.dallimoreGithub : damiendallimoreSplunkbase : damiendSlideshare : http://www.slideshare.net/damiendallimoreBlogs : http://blogs.splunk.com/devWeb : http://dev.splunk.com