Presentation section from Splunk Live content

1. Splunk Modular Inputs
Damien Dallimore
Developer Evangelist

2. Copyright©2013,SplunkInc.
Modular Inputs
2
• Extend the Splunk framework to define a custom input capability, just like the standard inputs you
are familiar with (TCP/UDP/File etc…)
• Splunk treats your custom input definitions as if they were part of Splunk's native inputs, totally
integrated first class citizen objects in Splunk
• Users interactively create and update your custom inputs using Splunk manager, just as they do for
native inputs. When deploying without a UI , you push out the inputs.conf file.
• All the properties are fully manageable via the REST API
• Version 5.0 +

3. Copyright©2013,SplunkInc.
What about scripted inputs ?
3
• Very loosely coupled to Splunk
• No standard configuration/schema framework
• No standard validation framework
• No standard lifecycle management
• Need to use “hacks” to make them running persistently
• Not really integrated with the REST API
• Logging not integrating with standard Splunk logs
BUT
• Their simplicity and loose coupling make them very rapid to develop
• Choose the right tool for the job

4. Copyright©2013,SplunkInc.
Diagram of Mod Input lifecycle
4
SplunkD
Init / Request Scheme
Mod Input
Return Scheme
External Validate
Confirm Validation
Execute
XML
XML
XML
Stream ResultsText /
XML
Validation
Code &
Error Msg
$SPLUNK_HOME/var/log/splunk/splunkd.log
logging

5. Copyright©2013,SplunkInc.
Scheme XML
5

6. Copyright©2013,SplunkInc.
Input XML
6
$SPLUNK_HOME/bin/splunk cmd splunkd print-modinput-config myscheme mystanza

7. Copyright©2013,SplunkInc.
Manage Mod Inputs via REST API
7

8. Copyright©2013,SplunkInc.
A few other technical features
8
• Validation
• External mode or via REST create/edit
• Run Mode
• single or multiple instance
• Checkpoint directory
• So your modular input can maintain state
• Streaming Mode
• Text or XML
• XML streaming has more syntactic sugar for meta data, event breaking
• Architecture specific scripts
• Splunk auto magically chooses the correct runtime script.

9. Copyright©2013,SplunkInc.
How are Mod Inputs going to help us
9
• We need to make it easy as possible to develop modular inputs , frameworks and tools
• Sometimes the greatest battle is just getting the data in , modular inputs are a great tool in our
armory.
• Bundle Modular Inputs in with the core product (DB, JMX, SNMP, JMS etc…)
• We need to make it easy to search for, install and configure these “data connectors”

10. Copyright©2013,SplunkInc.
Developing
10
• My preference is to use Python, however any language can be used.
• http://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/ModInputsIntro
• There is a certain amount of “plumbing” to put in place , so I like utilitys that take care of this for
you, so you can just focus on the business logic.
• I created utilitys to allow developers to rapidly create Modular Inputs in Java and Python
• https://github.com/damiendallimore/SplunkModularInputsJavaFramework
• https://github.com/damiendallimore/SplunkModularInputsPythonFramework
• HelloWorld examples to get you started
• Java -> JMS Messaging Modular Input , on Splunkbase
• Python -> SNMP Modular Input , soon to be released

11. Copyright©2013,SplunkInc.
Mod Inputs on Splunkbase
11

12. Copyright©2013,SplunkInc.
Messaging
12
• Message Oriented Middleware (MOM) infrastructures facilitate the sending/receiving of
messages between distributed systems
• Topics (publish/subscribe) and Queues (point to point)
• The glue that stitches heterogeneous enterprise computing environments together
• Represents a massive source of machine data that can be fed into Splunk to derive operational
visibility into your messaging environment and the various systems and applications that are
communicating via MOM

13. Copyright©2013,SplunkInc.
Building a Splunk Messaging Solution
13
• There has been considerable demand for functionality in Splunk to index messages
from queues/topics
• Ad hoc, proprietary, roll your own solutions were the only way
• I wanted to develop an integrated mechanism to allow Splunk users to connect to
their MOM and index their messages
• Modular Inputs provided the perfect platform to build a messaging solution

14. Copyright©2013,SplunkInc.
JMS Messaging Modular Input
14
• JMS is simply a messaging interface that abstracts your underlying MOM provider
implementation
• Most MOM vendors support JMS
• So this allowed for creating 1 single modular input that can index messages from :
• MQ Series / Websphere MQ
• Tibco EMS
• ActiveMQ
• HornetQ
• RabbitMQ
• SonicMQ
• JBoss Messaging
• Weblogic JMS
• Native JMS
• StormMQ
• MSMQ (with a bit of stuffing around)
• Etc…
• Simple to install : download from Splunkbase, drop in your apps directory, restart Splunk

15. Copyright©2013,SplunkInc.
Key Features
15
• Known to work with all aforementioned Messaging platforms
• Should work against any MOM platform with a JMS provider
• Runs on all supported Splunk platforms
• Consume messages from Topics and Queues
• Browse Queues (if you don’t want to consume the messages) and just Splunk
queue stats
• Messages header, properties and body indexed in Splunk in simple key/value
pairs
• Can plug in your own message handler if you require customized processing of
the message body
• Authentication and SSL support
• Scales horizontally if you require large volume message consumption

16. Copyright©2013,SplunkInc.
JMS input fully integrated into Splunk
16

17. Copyright©2013,SplunkInc.
Add a new queue/topic input
17

18. Copyright©2013,SplunkInc.
Configure the properties to connect
18

19. Copyright©2013,SplunkInc.
Get instant operational visibility
19

20. Demos
JMS (ActiveMQ , Websphere MQ)
SNMP

21. Copyright©2013,SplunkInc.
Contact me
21
Email : ddallimore@splunk.com
Twitter : @damiendallimore
Skype : damien.dallimore
Github : damiendallimore
Splunkbase : damiend
Slideshare : http://www.slideshare.net/damiendallimore
Blogs : http://blogs.splunk.com/dev
Web : http://dev.splunk.com