Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Splunk Modular Inputs / JMS Messaging Module Input


Published on

Presentation section from Splunk Live content

Published in: Technology
  • Dating for everyone is here: ❤❤❤ ❤❤❤
    Are you sure you want to  Yes  No
    Your message goes here
  • Sex in your area is here: ❤❤❤ ❤❤❤
    Are you sure you want to  Yes  No
    Your message goes here

Splunk Modular Inputs / JMS Messaging Module Input

  1. 1. Splunk Modular InputsDamien DallimoreDeveloper Evangelist
  2. 2. Copyright©2013,SplunkInc.Modular Inputs2• Extend the Splunk framework to define a custom input capability, just like the standard inputs youare familiar with (TCP/UDP/File etc…)• Splunk treats your custom input definitions as if they were part of Splunks native inputs, totallyintegrated first class citizen objects in Splunk• Users interactively create and update your custom inputs using Splunk manager, just as they do fornative inputs. When deploying without a UI , you push out the inputs.conf file.• All the properties are fully manageable via the REST API• Version 5.0 +
  3. 3. Copyright©2013,SplunkInc.What about scripted inputs ?3• Very loosely coupled to Splunk• No standard configuration/schema framework• No standard validation framework• No standard lifecycle management• Need to use “hacks” to make them running persistently• Not really integrated with the REST API• Logging not integrating with standard Splunk logsBUT• Their simplicity and loose coupling make them very rapid to develop• Choose the right tool for the job
  4. 4. Copyright©2013,SplunkInc.Diagram of Mod Input lifecycle4SplunkDInit / Request SchemeMod InputReturn SchemeExternal ValidateConfirm ValidationExecuteXMLXMLXMLStream ResultsText /XMLValidationCode &Error Msg$SPLUNK_HOME/var/log/splunk/splunkd.loglogging
  5. 5. Copyright©2013,SplunkInc.Scheme XML5
  6. 6. Copyright©2013,SplunkInc.Input XML6$SPLUNK_HOME/bin/splunk cmd splunkd print-modinput-config myscheme mystanza
  7. 7. Copyright©2013,SplunkInc.Manage Mod Inputs via REST API7
  8. 8. Copyright©2013,SplunkInc.A few other technical features8• Validation• External mode or via REST create/edit• Run Mode• single or multiple instance• Checkpoint directory• So your modular input can maintain state• Streaming Mode• Text or XML• XML streaming has more syntactic sugar for meta data, event breaking• Architecture specific scripts• Splunk auto magically chooses the correct runtime script.
  9. 9. Copyright©2013,SplunkInc.How are Mod Inputs going to help us9• We need to make it easy as possible to develop modular inputs , frameworks and tools• Sometimes the greatest battle is just getting the data in , modular inputs are a great tool in ourarmory.• Bundle Modular Inputs in with the core product (DB, JMX, SNMP, JMS etc…)• We need to make it easy to search for, install and configure these “data connectors”
  10. 10. Copyright©2013,SplunkInc.Developing10• My preference is to use Python, however any language can be used.•• There is a certain amount of “plumbing” to put in place , so I like utilitys that take care of this foryou, so you can just focus on the business logic.• I created utilitys to allow developers to rapidly create Modular Inputs in Java and Python••• HelloWorld examples to get you started• Java -> JMS Messaging Modular Input , on Splunkbase• Python -> SNMP Modular Input , soon to be released
  11. 11. Copyright©2013,SplunkInc.Mod Inputs on Splunkbase11
  12. 12. Copyright©2013,SplunkInc.Messaging12• Message Oriented Middleware (MOM) infrastructures facilitate the sending/receiving ofmessages between distributed systems• Topics (publish/subscribe) and Queues (point to point)• The glue that stitches heterogeneous enterprise computing environments together• Represents a massive source of machine data that can be fed into Splunk to derive operationalvisibility into your messaging environment and the various systems and applications that arecommunicating via MOM
  13. 13. Copyright©2013,SplunkInc.Building a Splunk Messaging Solution13• There has been considerable demand for functionality in Splunk to index messagesfrom queues/topics• Ad hoc, proprietary, roll your own solutions were the only way• I wanted to develop an integrated mechanism to allow Splunk users to connect totheir MOM and index their messages• Modular Inputs provided the perfect platform to build a messaging solution
  14. 14. Copyright©2013,SplunkInc.JMS Messaging Modular Input14• JMS is simply a messaging interface that abstracts your underlying MOM providerimplementation• Most MOM vendors support JMS• So this allowed for creating 1 single modular input that can index messages from :• MQ Series / Websphere MQ• Tibco EMS• ActiveMQ• HornetQ• RabbitMQ• SonicMQ• JBoss Messaging• Weblogic JMS• Native JMS• StormMQ• MSMQ (with a bit of stuffing around)• Etc…• Simple to install : download from Splunkbase, drop in your apps directory, restart Splunk
  15. 15. Copyright©2013,SplunkInc.Key Features15• Known to work with all aforementioned Messaging platforms• Should work against any MOM platform with a JMS provider• Runs on all supported Splunk platforms• Consume messages from Topics and Queues• Browse Queues (if you don’t want to consume the messages) and just Splunkqueue stats• Messages header, properties and body indexed in Splunk in simple key/valuepairs• Can plug in your own message handler if you require customized processing ofthe message body• Authentication and SSL support• Scales horizontally if you require large volume message consumption
  16. 16. Copyright©2013,SplunkInc.JMS input fully integrated into Splunk16
  17. 17. Copyright©2013,SplunkInc.Add a new queue/topic input17
  18. 18. Copyright©2013,SplunkInc.Configure the properties to connect18
  19. 19. Copyright©2013,SplunkInc.Get instant operational visibility19
  20. 20. DemosJMS (ActiveMQ , Websphere MQ)SNMP
  21. 21. Copyright©2013,SplunkInc.Contact me21Email : ddallimore@splunk.comTwitter : @damiendallimoreSkype : damien.dallimoreGithub : damiendallimoreSplunkbase : damiendSlideshare : : :