SlideShare a Scribd company logo
Copyright © 2014 Splunk Inc. 
Getting the Message 
Damien Dallimore 
Dev Evangelist , CSO Office @ Splunk 
Nimish Doshi 
Principal Systems Engineer @ Splunk
Disclaimer 
During the course of this presentation, we may make forward looking statements regarding future events or the 
expected performance of the company. We caution you that such statements reflect our current expectations and 
estimates based on factors currently known to us and that actual events or results could differ materially. For important 
factors that may cause actual results to differ from those contained in our forward-looking statements, please review 
our filings with the SEC. The forward-looking statements made in the this presentation are being made as of the time 
and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or 
accurate information. We do not assume any obligation to update any forward looking statements we may make. In 
addition, any information about our roadmap outlines our general product direction and is subject to change at any 
time without notice. It is for informational purposes only and shall not, be incorporated into any contract or other 
commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include 
any such feature or functionality in a future release. 
2
Agenda 
3 
Damien’s Section 
What is messaging 
JMS + Demo 
AMQP + Demo 
Kafka + Demo 
Custom message handling 
Architecting for scale 
Nimish’s Section 
Using ZeroMQ 
Using JMS for underutilized computers 
Question time
Damien’s Section
5 
From Middle Earth 
Make Splunk Apps & Add-ons 
Messaging background
6
apps.splunk.com 
github.com/damiendallimore 
7
What is messaging ? 
Messaging infrastructures facilitate the sending/receiving of messages between distributed systems 
Message can be encoded in one of many available protocols 
A common paradigm involves producers and consumers exchanging via topics or queues 
8 
Topics (publish subscribe) 
Queues (point to point) 
TOPIC 
QUEUE
Why are messaging architectures used ? 
Integrating Legacy Systems 
Integrating Heterogeneous Systems 
Distributed Applications 
Cluster Communication 
High Performance Streaming 
9
There’s a lot of information in the pipes 
10
The data opportunity 
Easily tap into a massive source of valuable inflight data flowing around the veins 
Don’t need to access the application directly ,pull data off the messaging bus 
I can not think of a single industry vertical that does not use messaging 
11
Getting this data into Splunk 
Many different messaging platforms and protocols 
JMS (Java Message Service) 
AMQP (Advanced Message Queueing Protocol) 
Kafka 
Nimish will cover some more uses cases also 
12
JMS 
Not a messaging protocol , but a programming interface to many different 
underlying message providers 
WebsphereMQ , Tibco EMS , ActiveMQ , HornetQ , SonicMQ etc… 
Very prevalent in the enterprise software landscape 
DEMO 
13
AMQP 
RabbitMQ 
Supports AMQP 0.9.1, 0.9, 0.8 
Common in financial services and environments that need high performance 
and low latency 
DEMO 
14
Kafka 
Cluster centric design = strong durability and fault tolerance 
Scales elastically 
Producers and Consumers communicate via topics in a Kafka node cluster 
Very popular with open source big data / streaming analytics solutions 
DEMO 
15
Custom message handling 
These Modular Inputs can be used in a multitude of scenarios 
Message bodies can be anything : JSON, XML, CSV, Unstructured text, Binary 
Need to give the end user the ability to customize message processing 
So you can plugin your own custom handlers 
Need to write code , but it is really easy , and there are examples on GitHub 
I’m a big data pre processing fan 
16
Cut the code 
17
Compile, bundle into jar file, copy to Splunk 
18
Declaratively apply it 
Let’s see if it works 
19
Achieving desired scale 
AMQP Mod Input 
AMQP Queue 
20 
Single Splunk Instance 
With 1 Modular Input instance , only so much performance / throughput can be achieved 
You’ll hit limits with JVM heap , CPU , OS STDIN/STDOUT Buffer , Splunk indexing pipeline
So go Horizontal 
AMQP Queue 
21 
Splunk Indexer Cluster 
Universal Forwarders 
AMQP Broker 
AMQP Mod Input AMQP Mod Input
Nimish’s Section
About Me 
• Principal Systems Engineer at Splunk in the NorthEast 
• Session Speaker at all past Splunk .conf user conferences 
• Catch me on the Splunk Blogs 
23
Problem with Getting Business Data from JMS 
The goal is to index the business message contents into Splunk 
Message Uncertainty Principal: 
If you de-queue the message to look at it, you have affected the TXN 
If you use various browse APIs for content, you may miss it 
– Message may have already been consumed by TXN 
Suggestion: Use a parallel queue to log the message 
– Suggestion: Try ZeroMQ 
24
Why use ZeroMQ 
Light Weight 
Multiple Client language support (Python, C++, Java, etc) 
Multiple design patterns (Pub/Sub, Pipeline, Request/Reply, etc) 
Open Source with community support 
25
Application Queue and ZeroMQ Example 
26 
Auto Load Balance 
1 
2
Example Python Sender 
context = zmq.Context() 
socket = context.socket(zmq.PUSH) 
socket.connect('tcp://127.0.0.1:5000') 
sleeptime=0.5 
27 
while True: 
num=random.randint(50,100) 
now = str(datetime.datetime.now()) 
sleep(sleeptime) 
payload = now + " Temperature=" + str(num) 
socket.send(payload)
Python Receiver (Scripted Input) 
context = zmq.Context() 
socket = context.socket(zmq.PULL) 
# Change address and port to match your environment 
socket.bind("tcp://127.0.0.1:5000") 
28 
while True: 
msg = socket.recv() 
print "%s" % msg 
except: 
print "exception"
Python Subscriber (Scripted Input) 
context = zmq.Context() 
socket = context.socket(zmq.SUB) 
socket.connect ("tcp://localhost:5556") 
# Subscribe to direction 
filter = "east" 
socket.setsockopt(zmq.SUBSCRIBE, filter) 
29 
while True: 
string = socket.recv() 
print string
Parallel Pipeline Example 
30
Getting Events out of Splunk 
31 
Splunk SDK 
Use Cases: 
– In Depth processing of Splunk events in a queued manner 
– Use as pivot point to drop off events into a Complex Event Processor 
– Batch Processing of Splunk events outside of Splunk 
 Divide and Conquer Approach as seen in last slide
Java Example using SDK to load ZeroMQ 
String query=search; 
Job job = service.getJobs().create(query, queryArgs); 
while (!job.isDone()) { 
32 
Thread.sleep(100); 
job.refresh(); 
} 
// Get Query Results and store in String str… (Code Omitted) 
// Assuming single line events 
StringTokenizer st = new StringTokenizer(str, "n"); 
while(st.hasMoreTokens()) { 
String temp= st.nextToken(); 
sock.send(temp.getBytes(), 0); 
byte response[] = sock.recv(0); 
}
Idle Computers at a Corporation 
33 
…
Idea: Use Ideas from SETI @ Home 
34
Idle Computers Put to Work Using JMS 
35 
…
Applications for Distributing Work 
Application Server would free up computing resources 
Work could be pushed to underutilized computers 
Examples: 
– Massive Mortgage Calculation Scenarios 
– Linear Optimization Problems 
– Matrix Multiplication 
– Compute all possible paths for combinatorics 
36
Architecture 
Optional 
37
Algorithm 
Application servers push requests to queues, which may include data 
in the request object called a Unit of Work 
JMS client implements doWork() interface to work with data 
Message Driven Bean receives finished work and implements 
doStore() interface 
What does this have to do with Splunk? 
– Time Series results can be stored in Splunk for further or historical analytics 
38
Matrix Example High Level Architecture 
39
Search Language Against Matrix Result 
List Column Values of Each Stored Multiplied Matrix using Multikv 
40 
Screenshot here
Search Language Against Matrix Result 
Visualize the Average for Columns 2 to 5 
41 
Screenshot here
Search Language Against Matrix Result 
Perform arbitrary math on aggregate columns 
42 
Screenshot here
Reference 
ZeroMQ 
– http://apps.splunk.com/app/1000/ 
– Blog: http://blogs.splunk.com/2012/06/08/zeromq-as-a-splunk-input/ 
Using JMS for Underutilized Computers 
– Github Reference: https://github.com/nimishdoshi/JMSClientApp/ 
– Blog: http://blogs.splunk.com/2014/04/11/splunk-as-a-recipient-on-the-jms-grid/ 
– Article:http://www.oracle.com/technetwork/articles/entarch/jms-distributed-work- 
082249.html 
43
Questions ?
THANK YOU 
ddallimore@splunk.com 
ndoshi@splunk.com

More Related Content

What's hot

How to build a Neutron Plugin (stadium edition)
How to build a Neutron Plugin (stadium edition)How to build a Neutron Plugin (stadium edition)
How to build a Neutron Plugin (stadium edition)
Salvatore Orlando
 
KubeFlow + GPU + Keras/TensorFlow 2.0 + TF Extended (TFX) + Kubernetes + PyTo...
KubeFlow + GPU + Keras/TensorFlow 2.0 + TF Extended (TFX) + Kubernetes + PyTo...KubeFlow + GPU + Keras/TensorFlow 2.0 + TF Extended (TFX) + Kubernetes + PyTo...
KubeFlow + GPU + Keras/TensorFlow 2.0 + TF Extended (TFX) + Kubernetes + PyTo...
Chris Fregly
 
Elk ruminating on logs
Elk ruminating on logsElk ruminating on logs
Elk ruminating on logs
Mathew Beane
 
OSMC 2021 | Robotmk: You don’t run IT – you deliver services!
OSMC 2021 | Robotmk: You don’t run IT – you deliver services!OSMC 2021 | Robotmk: You don’t run IT – you deliver services!
OSMC 2021 | Robotmk: You don’t run IT – you deliver services!
NETWAYS
 
Kubernetes Summit 2019 - Harden Your Kubernetes Cluster
Kubernetes Summit 2019 - Harden Your Kubernetes ClusterKubernetes Summit 2019 - Harden Your Kubernetes Cluster
Kubernetes Summit 2019 - Harden Your Kubernetes Cluster
smalltown
 
Performance Testing using Real Browsers with JMeter & Webdriver
Performance Testing using Real Browsers with JMeter & WebdriverPerformance Testing using Real Browsers with JMeter & Webdriver
Performance Testing using Real Browsers with JMeter & Webdriver
BlazeMeter
 
OpenStack Summit Vancouver: Lessons learned on upgrades
OpenStack Summit Vancouver:  Lessons learned on upgradesOpenStack Summit Vancouver:  Lessons learned on upgrades
OpenStack Summit Vancouver: Lessons learned on upgrades
Frédéric Lepied
 
Cloud: From Unmanned Data Center to Algorithmic Economy using Openstack
Cloud: From Unmanned Data Center to Algorithmic Economy using OpenstackCloud: From Unmanned Data Center to Algorithmic Economy using Openstack
Cloud: From Unmanned Data Center to Algorithmic Economy using Openstack
Andrew Yongjoon Kong
 
Performance Comparison of Streaming Big Data Platforms
Performance Comparison of Streaming Big Data PlatformsPerformance Comparison of Streaming Big Data Platforms
Performance Comparison of Streaming Big Data Platforms
DataWorks Summit/Hadoop Summit
 
OpenStack Tempest and REST API testing
OpenStack Tempest and REST API testingOpenStack Tempest and REST API testing
OpenStack Tempest and REST API testing
openstackindia
 
Automation Evolution with Junos
Automation Evolution with JunosAutomation Evolution with Junos
Automation Evolution with Junos
MarketingArrowECS_CZ
 
Securing your Pulsar Cluster with Vault_Chris Kellogg
Securing your Pulsar Cluster with Vault_Chris KelloggSecuring your Pulsar Cluster with Vault_Chris Kellogg
Securing your Pulsar Cluster with Vault_Chris Kellogg
StreamNative
 
So we're running Apache ZooKeeper. Now What? By Camille Fournier
So we're running Apache ZooKeeper. Now What? By Camille Fournier So we're running Apache ZooKeeper. Now What? By Camille Fournier
So we're running Apache ZooKeeper. Now What? By Camille Fournier
Hakka Labs
 
Software Defined Networking: The OpenDaylight Project
Software Defined Networking: The OpenDaylight ProjectSoftware Defined Networking: The OpenDaylight Project
Software Defined Networking: The OpenDaylight Project
Great Wide Open
 
Openwhisk - Colorado Meetups
Openwhisk - Colorado MeetupsOpenwhisk - Colorado Meetups
Openwhisk - Colorado Meetups
Upkar Lidder
 
Kafka Reliability - When it absolutely, positively has to be there
Kafka Reliability - When it absolutely, positively has to be thereKafka Reliability - When it absolutely, positively has to be there
Kafka Reliability - When it absolutely, positively has to be there
Gwen (Chen) Shapira
 
Kafka 0.8.0 Presentation to Atlanta Java User's Group March 2013
Kafka 0.8.0 Presentation to Atlanta Java User's Group March 2013Kafka 0.8.0 Presentation to Atlanta Java User's Group March 2013
Kafka 0.8.0 Presentation to Atlanta Java User's Group March 2013
Christopher Curtin
 
Topology Service Injection using Dragonflow & Kuryr
Topology Service Injection using Dragonflow & KuryrTopology Service Injection using Dragonflow & Kuryr
Topology Service Injection using Dragonflow & Kuryr
Eshed Gal-Or
 
Understanding and Extending Prometheus AlertManager
Understanding and Extending Prometheus AlertManagerUnderstanding and Extending Prometheus AlertManager
Understanding and Extending Prometheus AlertManager
Lee Calcote
 
Spark on Kubernetes - Advanced Spark and Tensorflow Meetup - Jan 19 2017 - An...
Spark on Kubernetes - Advanced Spark and Tensorflow Meetup - Jan 19 2017 - An...Spark on Kubernetes - Advanced Spark and Tensorflow Meetup - Jan 19 2017 - An...
Spark on Kubernetes - Advanced Spark and Tensorflow Meetup - Jan 19 2017 - An...
Chris Fregly
 

What's hot (20)

How to build a Neutron Plugin (stadium edition)
How to build a Neutron Plugin (stadium edition)How to build a Neutron Plugin (stadium edition)
How to build a Neutron Plugin (stadium edition)
 
KubeFlow + GPU + Keras/TensorFlow 2.0 + TF Extended (TFX) + Kubernetes + PyTo...
KubeFlow + GPU + Keras/TensorFlow 2.0 + TF Extended (TFX) + Kubernetes + PyTo...KubeFlow + GPU + Keras/TensorFlow 2.0 + TF Extended (TFX) + Kubernetes + PyTo...
KubeFlow + GPU + Keras/TensorFlow 2.0 + TF Extended (TFX) + Kubernetes + PyTo...
 
Elk ruminating on logs
Elk ruminating on logsElk ruminating on logs
Elk ruminating on logs
 
OSMC 2021 | Robotmk: You don’t run IT – you deliver services!
OSMC 2021 | Robotmk: You don’t run IT – you deliver services!OSMC 2021 | Robotmk: You don’t run IT – you deliver services!
OSMC 2021 | Robotmk: You don’t run IT – you deliver services!
 
Kubernetes Summit 2019 - Harden Your Kubernetes Cluster
Kubernetes Summit 2019 - Harden Your Kubernetes ClusterKubernetes Summit 2019 - Harden Your Kubernetes Cluster
Kubernetes Summit 2019 - Harden Your Kubernetes Cluster
 
Performance Testing using Real Browsers with JMeter & Webdriver
Performance Testing using Real Browsers with JMeter & WebdriverPerformance Testing using Real Browsers with JMeter & Webdriver
Performance Testing using Real Browsers with JMeter & Webdriver
 
OpenStack Summit Vancouver: Lessons learned on upgrades
OpenStack Summit Vancouver:  Lessons learned on upgradesOpenStack Summit Vancouver:  Lessons learned on upgrades
OpenStack Summit Vancouver: Lessons learned on upgrades
 
Cloud: From Unmanned Data Center to Algorithmic Economy using Openstack
Cloud: From Unmanned Data Center to Algorithmic Economy using OpenstackCloud: From Unmanned Data Center to Algorithmic Economy using Openstack
Cloud: From Unmanned Data Center to Algorithmic Economy using Openstack
 
Performance Comparison of Streaming Big Data Platforms
Performance Comparison of Streaming Big Data PlatformsPerformance Comparison of Streaming Big Data Platforms
Performance Comparison of Streaming Big Data Platforms
 
OpenStack Tempest and REST API testing
OpenStack Tempest and REST API testingOpenStack Tempest and REST API testing
OpenStack Tempest and REST API testing
 
Automation Evolution with Junos
Automation Evolution with JunosAutomation Evolution with Junos
Automation Evolution with Junos
 
Securing your Pulsar Cluster with Vault_Chris Kellogg
Securing your Pulsar Cluster with Vault_Chris KelloggSecuring your Pulsar Cluster with Vault_Chris Kellogg
Securing your Pulsar Cluster with Vault_Chris Kellogg
 
So we're running Apache ZooKeeper. Now What? By Camille Fournier
So we're running Apache ZooKeeper. Now What? By Camille Fournier So we're running Apache ZooKeeper. Now What? By Camille Fournier
So we're running Apache ZooKeeper. Now What? By Camille Fournier
 
Software Defined Networking: The OpenDaylight Project
Software Defined Networking: The OpenDaylight ProjectSoftware Defined Networking: The OpenDaylight Project
Software Defined Networking: The OpenDaylight Project
 
Openwhisk - Colorado Meetups
Openwhisk - Colorado MeetupsOpenwhisk - Colorado Meetups
Openwhisk - Colorado Meetups
 
Kafka Reliability - When it absolutely, positively has to be there
Kafka Reliability - When it absolutely, positively has to be thereKafka Reliability - When it absolutely, positively has to be there
Kafka Reliability - When it absolutely, positively has to be there
 
Kafka 0.8.0 Presentation to Atlanta Java User's Group March 2013
Kafka 0.8.0 Presentation to Atlanta Java User's Group March 2013Kafka 0.8.0 Presentation to Atlanta Java User's Group March 2013
Kafka 0.8.0 Presentation to Atlanta Java User's Group March 2013
 
Topology Service Injection using Dragonflow & Kuryr
Topology Service Injection using Dragonflow & KuryrTopology Service Injection using Dragonflow & Kuryr
Topology Service Injection using Dragonflow & Kuryr
 
Understanding and Extending Prometheus AlertManager
Understanding and Extending Prometheus AlertManagerUnderstanding and Extending Prometheus AlertManager
Understanding and Extending Prometheus AlertManager
 
Spark on Kubernetes - Advanced Spark and Tensorflow Meetup - Jan 19 2017 - An...
Spark on Kubernetes - Advanced Spark and Tensorflow Meetup - Jan 19 2017 - An...Spark on Kubernetes - Advanced Spark and Tensorflow Meetup - Jan 19 2017 - An...
Spark on Kubernetes - Advanced Spark and Tensorflow Meetup - Jan 19 2017 - An...
 

Similar to Splunk Conf 2014 - Getting the message

Building and deploying LLM applications with Apache Airflow
Building and deploying LLM applications with Apache AirflowBuilding and deploying LLM applications with Apache Airflow
Building and deploying LLM applications with Apache Airflow
Kaxil Naik
 
Apache Beam: A unified model for batch and stream processing data
Apache Beam: A unified model for batch and stream processing dataApache Beam: A unified model for batch and stream processing data
Apache Beam: A unified model for batch and stream processing data
DataWorks Summit/Hadoop Summit
 
VWBPE 2020 - Overcoming LSL Limitations in Second Life
VWBPE 2020 - Overcoming LSL Limitations in Second LifeVWBPE 2020 - Overcoming LSL Limitations in Second Life
VWBPE 2020 - Overcoming LSL Limitations in Second Life
jbhancroft
 
Implementing Messaging Patterns in JavaScript using the OpenAjax Hub
Implementing Messaging Patterns in JavaScript using the OpenAjax HubImplementing Messaging Patterns in JavaScript using the OpenAjax Hub
Implementing Messaging Patterns in JavaScript using the OpenAjax Hub
Kevin Hakanson
 
Open Source XMPP for Cloud Services
Open Source XMPP for Cloud ServicesOpen Source XMPP for Cloud Services
Open Source XMPP for Cloud Services
mattjive
 
DDS Advanced Tutorial - OMG June 2013 Berlin Meeting
DDS Advanced Tutorial - OMG June 2013 Berlin MeetingDDS Advanced Tutorial - OMG June 2013 Berlin Meeting
DDS Advanced Tutorial - OMG June 2013 Berlin Meeting
Jaime Martin Losa
 
Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions
Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure FunctionsMessaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions
Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions
John Staveley
 
moveMountainIEEE
moveMountainIEEEmoveMountainIEEE
moveMountainIEEE
Christopher Gallo
 
WSO2 Complex Event Processor - Product Overview
WSO2 Complex Event Processor - Product OverviewWSO2 Complex Event Processor - Product Overview
WSO2 Complex Event Processor - Product Overview
WSO2
 
Let’s Monitor Conditions at the Conference With Timothy Spann & David Kjerrum...
Let’s Monitor Conditions at the Conference With Timothy Spann & David Kjerrum...Let’s Monitor Conditions at the Conference With Timothy Spann & David Kjerrum...
Let’s Monitor Conditions at the Conference With Timothy Spann & David Kjerrum...
HostedbyConfluent
 
(Current22) Let's Monitor The Conditions at the Conference
(Current22) Let's Monitor The Conditions at the Conference(Current22) Let's Monitor The Conditions at the Conference
(Current22) Let's Monitor The Conditions at the Conference
Timothy Spann
 
Scaling Streaming - Concepts, Research, Goals
Scaling Streaming - Concepts, Research, GoalsScaling Streaming - Concepts, Research, Goals
Scaling Streaming - Concepts, Research, Goals
kamaelian
 
pythonOCC PDE2009 presentation
pythonOCC PDE2009 presentationpythonOCC PDE2009 presentation
pythonOCC PDE2009 presentation
Thomas Paviot
 
Distributed Systems: How to connect your real-time applications
Distributed Systems: How to connect your real-time applicationsDistributed Systems: How to connect your real-time applications
Distributed Systems: How to connect your real-time applications
Jaime Martin Losa
 
Running Accurate, Scalable, and Reproducible Simulations of Distributed Syste...
Running Accurate, Scalable, and Reproducible Simulations of Distributed Syste...Running Accurate, Scalable, and Reproducible Simulations of Distributed Syste...
Running Accurate, Scalable, and Reproducible Simulations of Distributed Syste...
Rafael Ferreira da Silva
 
Cloud C
Cloud CCloud C
Splunk Discovery: Warsaw 2018 - Getting Data In
Splunk Discovery: Warsaw 2018 - Getting Data InSplunk Discovery: Warsaw 2018 - Getting Data In
Splunk Discovery: Warsaw 2018 - Getting Data In
Splunk
 
Being HAPI! Reverse Proxying on Purpose
Being HAPI! Reverse Proxying on PurposeBeing HAPI! Reverse Proxying on Purpose
Being HAPI! Reverse Proxying on Purpose
Aman Kohli
 
Prometheus Training
Prometheus TrainingPrometheus Training
Prometheus Training
Tim Tyler
 
Forecast 2014: TOSCA Proof of Concept
Forecast 2014: TOSCA Proof of ConceptForecast 2014: TOSCA Proof of Concept
Forecast 2014: TOSCA Proof of Concept
Open Data Center Alliance
 

Similar to Splunk Conf 2014 - Getting the message (20)

Building and deploying LLM applications with Apache Airflow
Building and deploying LLM applications with Apache AirflowBuilding and deploying LLM applications with Apache Airflow
Building and deploying LLM applications with Apache Airflow
 
Apache Beam: A unified model for batch and stream processing data
Apache Beam: A unified model for batch and stream processing dataApache Beam: A unified model for batch and stream processing data
Apache Beam: A unified model for batch and stream processing data
 
VWBPE 2020 - Overcoming LSL Limitations in Second Life
VWBPE 2020 - Overcoming LSL Limitations in Second LifeVWBPE 2020 - Overcoming LSL Limitations in Second Life
VWBPE 2020 - Overcoming LSL Limitations in Second Life
 
Implementing Messaging Patterns in JavaScript using the OpenAjax Hub
Implementing Messaging Patterns in JavaScript using the OpenAjax HubImplementing Messaging Patterns in JavaScript using the OpenAjax Hub
Implementing Messaging Patterns in JavaScript using the OpenAjax Hub
 
Open Source XMPP for Cloud Services
Open Source XMPP for Cloud ServicesOpen Source XMPP for Cloud Services
Open Source XMPP for Cloud Services
 
DDS Advanced Tutorial - OMG June 2013 Berlin Meeting
DDS Advanced Tutorial - OMG June 2013 Berlin MeetingDDS Advanced Tutorial - OMG June 2013 Berlin Meeting
DDS Advanced Tutorial - OMG June 2013 Berlin Meeting
 
Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions
Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure FunctionsMessaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions
Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions
 
moveMountainIEEE
moveMountainIEEEmoveMountainIEEE
moveMountainIEEE
 
WSO2 Complex Event Processor - Product Overview
WSO2 Complex Event Processor - Product OverviewWSO2 Complex Event Processor - Product Overview
WSO2 Complex Event Processor - Product Overview
 
Let’s Monitor Conditions at the Conference With Timothy Spann & David Kjerrum...
Let’s Monitor Conditions at the Conference With Timothy Spann & David Kjerrum...Let’s Monitor Conditions at the Conference With Timothy Spann & David Kjerrum...
Let’s Monitor Conditions at the Conference With Timothy Spann & David Kjerrum...
 
(Current22) Let's Monitor The Conditions at the Conference
(Current22) Let's Monitor The Conditions at the Conference(Current22) Let's Monitor The Conditions at the Conference
(Current22) Let's Monitor The Conditions at the Conference
 
Scaling Streaming - Concepts, Research, Goals
Scaling Streaming - Concepts, Research, GoalsScaling Streaming - Concepts, Research, Goals
Scaling Streaming - Concepts, Research, Goals
 
pythonOCC PDE2009 presentation
pythonOCC PDE2009 presentationpythonOCC PDE2009 presentation
pythonOCC PDE2009 presentation
 
Distributed Systems: How to connect your real-time applications
Distributed Systems: How to connect your real-time applicationsDistributed Systems: How to connect your real-time applications
Distributed Systems: How to connect your real-time applications
 
Running Accurate, Scalable, and Reproducible Simulations of Distributed Syste...
Running Accurate, Scalable, and Reproducible Simulations of Distributed Syste...Running Accurate, Scalable, and Reproducible Simulations of Distributed Syste...
Running Accurate, Scalable, and Reproducible Simulations of Distributed Syste...
 
Cloud C
Cloud CCloud C
Cloud C
 
Splunk Discovery: Warsaw 2018 - Getting Data In
Splunk Discovery: Warsaw 2018 - Getting Data InSplunk Discovery: Warsaw 2018 - Getting Data In
Splunk Discovery: Warsaw 2018 - Getting Data In
 
Being HAPI! Reverse Proxying on Purpose
Being HAPI! Reverse Proxying on PurposeBeing HAPI! Reverse Proxying on Purpose
Being HAPI! Reverse Proxying on Purpose
 
Prometheus Training
Prometheus TrainingPrometheus Training
Prometheus Training
 
Forecast 2014: TOSCA Proof of Concept
Forecast 2014: TOSCA Proof of ConceptForecast 2014: TOSCA Proof of Concept
Forecast 2014: TOSCA Proof of Concept
 

More from Damien Dallimore

QCon London 2015 - Wrangling Data at the IOT Rodeo
QCon London 2015 - Wrangling Data at the IOT RodeoQCon London 2015 - Wrangling Data at the IOT Rodeo
QCon London 2015 - Wrangling Data at the IOT Rodeo
Damien Dallimore
 
SpringOne2GX 2014 Splunk Presentation
SpringOne2GX 2014 Splunk PresentationSpringOne2GX 2014 Splunk Presentation
SpringOne2GX 2014 Splunk Presentation
Damien Dallimore
 
SplunkLive London 2014 Developer Presentation
SplunkLive London 2014  Developer PresentationSplunkLive London 2014  Developer Presentation
SplunkLive London 2014 Developer Presentation
Damien Dallimore
 
A Brief History Of Data
A Brief History Of DataA Brief History Of Data
A Brief History Of Data
Damien Dallimore
 
Integrating Splunk into your Spring Applications
Integrating Splunk into your Spring ApplicationsIntegrating Splunk into your Spring Applications
Integrating Splunk into your Spring Applications
Damien Dallimore
 
Spring Integration Splunk
Spring Integration SplunkSpring Integration Splunk
Spring Integration Splunk
Damien Dallimore
 
Splunk Modular Inputs / JMS Messaging Module Input
Splunk Modular Inputs / JMS Messaging Module InputSplunk Modular Inputs / JMS Messaging Module Input
Splunk Modular Inputs / JMS Messaging Module Input
Damien Dallimore
 
Splunk Java Agent
Splunk Java AgentSplunk Java Agent
Splunk Java Agent
Damien Dallimore
 
Splunk Developer Platform
Splunk Developer PlatformSplunk Developer Platform
Splunk Developer Platform
Damien Dallimore
 
Splunk as a_big_data_platform_for_developers_spring_one2gx
Splunk as a_big_data_platform_for_developers_spring_one2gxSplunk as a_big_data_platform_for_developers_spring_one2gx
Splunk as a_big_data_platform_for_developers_spring_one2gx
Damien Dallimore
 
Using the Splunk Java SDK
Using the Splunk Java SDKUsing the Splunk Java SDK
Using the Splunk Java SDK
Damien Dallimore
 
Splunking the JVM (Java Virtual Machine)
Splunking the JVM (Java Virtual Machine)Splunking the JVM (Java Virtual Machine)
Splunking the JVM (Java Virtual Machine)
Damien Dallimore
 

More from Damien Dallimore (12)

QCon London 2015 - Wrangling Data at the IOT Rodeo
QCon London 2015 - Wrangling Data at the IOT RodeoQCon London 2015 - Wrangling Data at the IOT Rodeo
QCon London 2015 - Wrangling Data at the IOT Rodeo
 
SpringOne2GX 2014 Splunk Presentation
SpringOne2GX 2014 Splunk PresentationSpringOne2GX 2014 Splunk Presentation
SpringOne2GX 2014 Splunk Presentation
 
SplunkLive London 2014 Developer Presentation
SplunkLive London 2014  Developer PresentationSplunkLive London 2014  Developer Presentation
SplunkLive London 2014 Developer Presentation
 
A Brief History Of Data
A Brief History Of DataA Brief History Of Data
A Brief History Of Data
 
Integrating Splunk into your Spring Applications
Integrating Splunk into your Spring ApplicationsIntegrating Splunk into your Spring Applications
Integrating Splunk into your Spring Applications
 
Spring Integration Splunk
Spring Integration SplunkSpring Integration Splunk
Spring Integration Splunk
 
Splunk Modular Inputs / JMS Messaging Module Input
Splunk Modular Inputs / JMS Messaging Module InputSplunk Modular Inputs / JMS Messaging Module Input
Splunk Modular Inputs / JMS Messaging Module Input
 
Splunk Java Agent
Splunk Java AgentSplunk Java Agent
Splunk Java Agent
 
Splunk Developer Platform
Splunk Developer PlatformSplunk Developer Platform
Splunk Developer Platform
 
Splunk as a_big_data_platform_for_developers_spring_one2gx
Splunk as a_big_data_platform_for_developers_spring_one2gxSplunk as a_big_data_platform_for_developers_spring_one2gx
Splunk as a_big_data_platform_for_developers_spring_one2gx
 
Using the Splunk Java SDK
Using the Splunk Java SDKUsing the Splunk Java SDK
Using the Splunk Java SDK
 
Splunking the JVM (Java Virtual Machine)
Splunking the JVM (Java Virtual Machine)Splunking the JVM (Java Virtual Machine)
Splunking the JVM (Java Virtual Machine)
 

Recently uploaded

E-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet DynamicsE-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet Dynamics
Hornet Dynamics
 
How to write a program in any programming language
How to write a program in any programming languageHow to write a program in any programming language
How to write a program in any programming language
Rakesh Kumar R
 
Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)
Julian Hyde
 
WWDC 2024 Keynote Review: For CocoaCoders Austin
WWDC 2024 Keynote Review: For CocoaCoders AustinWWDC 2024 Keynote Review: For CocoaCoders Austin
WWDC 2024 Keynote Review: For CocoaCoders Austin
Patrick Weigel
 
Microservice Teams - How the cloud changes the way we work
Microservice Teams - How the cloud changes the way we workMicroservice Teams - How the cloud changes the way we work
Microservice Teams - How the cloud changes the way we work
Sven Peters
 
Lecture 2 - software testing SE 412.pptx
Lecture 2 - software testing SE 412.pptxLecture 2 - software testing SE 412.pptx
Lecture 2 - software testing SE 412.pptx
TaghreedAltamimi
 
Enums On Steroids - let's look at sealed classes !
Enums On Steroids - let's look at sealed classes !Enums On Steroids - let's look at sealed classes !
Enums On Steroids - let's look at sealed classes !
Marcin Chrost
 
Transform Your Communication with Cloud-Based IVR Solutions
Transform Your Communication with Cloud-Based IVR SolutionsTransform Your Communication with Cloud-Based IVR Solutions
Transform Your Communication with Cloud-Based IVR Solutions
TheSMSPoint
 
Oracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptxOracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptx
Remote DBA Services
 
Requirement Traceability in Xen Functional Safety
Requirement Traceability in Xen Functional SafetyRequirement Traceability in Xen Functional Safety
Requirement Traceability in Xen Functional Safety
Ayan Halder
 
一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理
dakas1
 
Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...
Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...
Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...
XfilesPro
 
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling ExtensionsUI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
Peter Muessig
 
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdfTop Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
VALiNTRY360
 
2024 eCommerceDays Toulouse - Sylius 2.0.pdf
2024 eCommerceDays Toulouse - Sylius 2.0.pdf2024 eCommerceDays Toulouse - Sylius 2.0.pdf
2024 eCommerceDays Toulouse - Sylius 2.0.pdf
Łukasz Chruściel
 
Artificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension FunctionsArtificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension Functions
Octavian Nadolu
 
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian CompaniesE-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
Quickdice ERP
 
How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?
ToXSL Technologies
 
Using Xen Hypervisor for Functional Safety
Using Xen Hypervisor for Functional SafetyUsing Xen Hypervisor for Functional Safety
Using Xen Hypervisor for Functional Safety
Ayan Halder
 
socradar-q1-2024-aviation-industry-report.pdf
socradar-q1-2024-aviation-industry-report.pdfsocradar-q1-2024-aviation-industry-report.pdf
socradar-q1-2024-aviation-industry-report.pdf
SOCRadar
 

Recently uploaded (20)

E-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet DynamicsE-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet Dynamics
 
How to write a program in any programming language
How to write a program in any programming languageHow to write a program in any programming language
How to write a program in any programming language
 
Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)
 
WWDC 2024 Keynote Review: For CocoaCoders Austin
WWDC 2024 Keynote Review: For CocoaCoders AustinWWDC 2024 Keynote Review: For CocoaCoders Austin
WWDC 2024 Keynote Review: For CocoaCoders Austin
 
Microservice Teams - How the cloud changes the way we work
Microservice Teams - How the cloud changes the way we workMicroservice Teams - How the cloud changes the way we work
Microservice Teams - How the cloud changes the way we work
 
Lecture 2 - software testing SE 412.pptx
Lecture 2 - software testing SE 412.pptxLecture 2 - software testing SE 412.pptx
Lecture 2 - software testing SE 412.pptx
 
Enums On Steroids - let's look at sealed classes !
Enums On Steroids - let's look at sealed classes !Enums On Steroids - let's look at sealed classes !
Enums On Steroids - let's look at sealed classes !
 
Transform Your Communication with Cloud-Based IVR Solutions
Transform Your Communication with Cloud-Based IVR SolutionsTransform Your Communication with Cloud-Based IVR Solutions
Transform Your Communication with Cloud-Based IVR Solutions
 
Oracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptxOracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptx
 
Requirement Traceability in Xen Functional Safety
Requirement Traceability in Xen Functional SafetyRequirement Traceability in Xen Functional Safety
Requirement Traceability in Xen Functional Safety
 
一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理
 
Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...
Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...
Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...
 
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling ExtensionsUI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
 
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdfTop Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
 
2024 eCommerceDays Toulouse - Sylius 2.0.pdf
2024 eCommerceDays Toulouse - Sylius 2.0.pdf2024 eCommerceDays Toulouse - Sylius 2.0.pdf
2024 eCommerceDays Toulouse - Sylius 2.0.pdf
 
Artificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension FunctionsArtificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension Functions
 
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian CompaniesE-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
 
How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?
 
Using Xen Hypervisor for Functional Safety
Using Xen Hypervisor for Functional SafetyUsing Xen Hypervisor for Functional Safety
Using Xen Hypervisor for Functional Safety
 
socradar-q1-2024-aviation-industry-report.pdf
socradar-q1-2024-aviation-industry-report.pdfsocradar-q1-2024-aviation-industry-report.pdf
socradar-q1-2024-aviation-industry-report.pdf
 

Splunk Conf 2014 - Getting the message

  • 1. Copyright © 2014 Splunk Inc. Getting the Message Damien Dallimore Dev Evangelist , CSO Office @ Splunk Nimish Doshi Principal Systems Engineer @ Splunk
  • 2. Disclaimer During the course of this presentation, we may make forward looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in the this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release. 2
  • 3. Agenda 3 Damien’s Section What is messaging JMS + Demo AMQP + Demo Kafka + Demo Custom message handling Architecting for scale Nimish’s Section Using ZeroMQ Using JMS for underutilized computers Question time
  • 5. 5 From Middle Earth Make Splunk Apps & Add-ons Messaging background
  • 6. 6
  • 8. What is messaging ? Messaging infrastructures facilitate the sending/receiving of messages between distributed systems Message can be encoded in one of many available protocols A common paradigm involves producers and consumers exchanging via topics or queues 8 Topics (publish subscribe) Queues (point to point) TOPIC QUEUE
  • 9. Why are messaging architectures used ? Integrating Legacy Systems Integrating Heterogeneous Systems Distributed Applications Cluster Communication High Performance Streaming 9
  • 10. There’s a lot of information in the pipes 10
  • 11. The data opportunity Easily tap into a massive source of valuable inflight data flowing around the veins Don’t need to access the application directly ,pull data off the messaging bus I can not think of a single industry vertical that does not use messaging 11
  • 12. Getting this data into Splunk Many different messaging platforms and protocols JMS (Java Message Service) AMQP (Advanced Message Queueing Protocol) Kafka Nimish will cover some more uses cases also 12
  • 13. JMS Not a messaging protocol , but a programming interface to many different underlying message providers WebsphereMQ , Tibco EMS , ActiveMQ , HornetQ , SonicMQ etc… Very prevalent in the enterprise software landscape DEMO 13
  • 14. AMQP RabbitMQ Supports AMQP 0.9.1, 0.9, 0.8 Common in financial services and environments that need high performance and low latency DEMO 14
  • 15. Kafka Cluster centric design = strong durability and fault tolerance Scales elastically Producers and Consumers communicate via topics in a Kafka node cluster Very popular with open source big data / streaming analytics solutions DEMO 15
  • 16. Custom message handling These Modular Inputs can be used in a multitude of scenarios Message bodies can be anything : JSON, XML, CSV, Unstructured text, Binary Need to give the end user the ability to customize message processing So you can plugin your own custom handlers Need to write code , but it is really easy , and there are examples on GitHub I’m a big data pre processing fan 16
  • 18. Compile, bundle into jar file, copy to Splunk 18
  • 19. Declaratively apply it Let’s see if it works 19
  • 20. Achieving desired scale AMQP Mod Input AMQP Queue 20 Single Splunk Instance With 1 Modular Input instance , only so much performance / throughput can be achieved You’ll hit limits with JVM heap , CPU , OS STDIN/STDOUT Buffer , Splunk indexing pipeline
  • 21. So go Horizontal AMQP Queue 21 Splunk Indexer Cluster Universal Forwarders AMQP Broker AMQP Mod Input AMQP Mod Input
  • 23. About Me • Principal Systems Engineer at Splunk in the NorthEast • Session Speaker at all past Splunk .conf user conferences • Catch me on the Splunk Blogs 23
  • 24. Problem with Getting Business Data from JMS The goal is to index the business message contents into Splunk Message Uncertainty Principal: If you de-queue the message to look at it, you have affected the TXN If you use various browse APIs for content, you may miss it – Message may have already been consumed by TXN Suggestion: Use a parallel queue to log the message – Suggestion: Try ZeroMQ 24
  • 25. Why use ZeroMQ Light Weight Multiple Client language support (Python, C++, Java, etc) Multiple design patterns (Pub/Sub, Pipeline, Request/Reply, etc) Open Source with community support 25
  • 26. Application Queue and ZeroMQ Example 26 Auto Load Balance 1 2
  • 27. Example Python Sender context = zmq.Context() socket = context.socket(zmq.PUSH) socket.connect('tcp://127.0.0.1:5000') sleeptime=0.5 27 while True: num=random.randint(50,100) now = str(datetime.datetime.now()) sleep(sleeptime) payload = now + " Temperature=" + str(num) socket.send(payload)
  • 28. Python Receiver (Scripted Input) context = zmq.Context() socket = context.socket(zmq.PULL) # Change address and port to match your environment socket.bind("tcp://127.0.0.1:5000") 28 while True: msg = socket.recv() print "%s" % msg except: print "exception"
  • 29. Python Subscriber (Scripted Input) context = zmq.Context() socket = context.socket(zmq.SUB) socket.connect ("tcp://localhost:5556") # Subscribe to direction filter = "east" socket.setsockopt(zmq.SUBSCRIBE, filter) 29 while True: string = socket.recv() print string
  • 31. Getting Events out of Splunk 31 Splunk SDK Use Cases: – In Depth processing of Splunk events in a queued manner – Use as pivot point to drop off events into a Complex Event Processor – Batch Processing of Splunk events outside of Splunk  Divide and Conquer Approach as seen in last slide
  • 32. Java Example using SDK to load ZeroMQ String query=search; Job job = service.getJobs().create(query, queryArgs); while (!job.isDone()) { 32 Thread.sleep(100); job.refresh(); } // Get Query Results and store in String str… (Code Omitted) // Assuming single line events StringTokenizer st = new StringTokenizer(str, "n"); while(st.hasMoreTokens()) { String temp= st.nextToken(); sock.send(temp.getBytes(), 0); byte response[] = sock.recv(0); }
  • 33. Idle Computers at a Corporation 33 …
  • 34. Idea: Use Ideas from SETI @ Home 34
  • 35. Idle Computers Put to Work Using JMS 35 …
  • 36. Applications for Distributing Work Application Server would free up computing resources Work could be pushed to underutilized computers Examples: – Massive Mortgage Calculation Scenarios – Linear Optimization Problems – Matrix Multiplication – Compute all possible paths for combinatorics 36
  • 38. Algorithm Application servers push requests to queues, which may include data in the request object called a Unit of Work JMS client implements doWork() interface to work with data Message Driven Bean receives finished work and implements doStore() interface What does this have to do with Splunk? – Time Series results can be stored in Splunk for further or historical analytics 38
  • 39. Matrix Example High Level Architecture 39
  • 40. Search Language Against Matrix Result List Column Values of Each Stored Multiplied Matrix using Multikv 40 Screenshot here
  • 41. Search Language Against Matrix Result Visualize the Average for Columns 2 to 5 41 Screenshot here
  • 42. Search Language Against Matrix Result Perform arbitrary math on aggregate columns 42 Screenshot here
  • 43. Reference ZeroMQ – http://apps.splunk.com/app/1000/ – Blog: http://blogs.splunk.com/2012/06/08/zeromq-as-a-splunk-input/ Using JMS for Underutilized Computers – Github Reference: https://github.com/nimishdoshi/JMSClientApp/ – Blog: http://blogs.splunk.com/2014/04/11/splunk-as-a-recipient-on-the-jms-grid/ – Article:http://www.oracle.com/technetwork/articles/entarch/jms-distributed-work- 082249.html 43
  • 45. THANK YOU ddallimore@splunk.com ndoshi@splunk.com

Editor's Notes

  1. From Auckland Dev evang , ex customer 5th Conf Make Apps , Cut code Through messaging background , a lot of integration work in many different industrys , particularly in the enterprise Java space.
  2. Everything 100% open source use , reuse , whatever. Collaborate Community answers.splunk.com for support is best
  3. Enterprise Service Buses Multi tier apps ,asynch processing Apache Storm That pretty broadly covers most enterprise software scenarios.
  4. Interoperablity not guaranteed message producers and consumers may be implemented differently You “plugin” the underlying message provider implemention
  5. Wire level protocol, hence better interoperabilty than JMS and better performance Usual messaging features such as , Flow control , guaranteed delivery, quality of service etc… JP Morgan chase 1.0 is an entirely different protocol , any demand for this ?? Swiftmq Apache apollo Apache qpid
  6. Manage access to the cluster with Apache Zookeeper Data streams can be partitioned over multiple machines in the cluster Apache storm spout
  7. If you have to opportunity to get the data into an optimal format for Splunk , do it. Handle custom payloads , even binary Efficient use of license Pre compute some values that might not be best suited to the Splunk search language
  8. Inputting the setting into stanza Send message Show reversed output
  9. com.splunk.modinput.jms.customhandler.MessageTextReverser
  10. Same pattern applys to JMS and Kafka
  11. Your only limits are going to be your ability to provision Splunk nodes. Same pattern applys to other Mod Inputs Works with queues , not pub sub topics (you’ll get duplicates)