Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Copyright © 2014 Splunk Inc. 
Getting the Message 
Damien Dallimore 
Dev Evangelist , CSO Office @ Splunk 
Nimish Doshi 
P...
Disclaimer 
During the course of this presentation, we may make forward looking statements regarding future events or the ...
Agenda 
3 
Damien’s Section 
What is messaging 
JMS + Demo 
AMQP + Demo 
Kafka + Demo 
Custom message handling 
Architecti...
Damien’s Section
5 
From Middle Earth 
Make Splunk Apps & Add-ons 
Messaging background
6
apps.splunk.com 
github.com/damiendallimore 
7
What is messaging ? 
Messaging infrastructures facilitate the sending/receiving of messages between distributed systems 
M...
Why are messaging architectures used ? 
Integrating Legacy Systems 
Integrating Heterogeneous Systems 
Distributed Applica...
There’s a lot of information in the pipes 
10
The data opportunity 
Easily tap into a massive source of valuable inflight data flowing around the veins 
Don’t need to a...
Getting this data into Splunk 
Many different messaging platforms and protocols 
JMS (Java Message Service) 
AMQP (Advance...
JMS 
Not a messaging protocol , but a programming interface to many different 
underlying message providers 
WebsphereMQ ,...
AMQP 
RabbitMQ 
Supports AMQP 0.9.1, 0.9, 0.8 
Common in financial services and environments that need high performance 
a...
Kafka 
Cluster centric design = strong durability and fault tolerance 
Scales elastically 
Producers and Consumers communi...
Custom message handling 
These Modular Inputs can be used in a multitude of scenarios 
Message bodies can be anything : JS...
Cut the code 
17
Compile, bundle into jar file, copy to Splunk 
18
Declaratively apply it 
Let’s see if it works 
19
Achieving desired scale 
AMQP Mod Input 
AMQP Queue 
20 
Single Splunk Instance 
With 1 Modular Input instance , only so m...
So go Horizontal 
AMQP Queue 
21 
Splunk Indexer Cluster 
Universal Forwarders 
AMQP Broker 
AMQP Mod Input AMQP Mod Input
Nimish’s Section
About Me 
• Principal Systems Engineer at Splunk in the NorthEast 
• Session Speaker at all past Splunk .conf user confere...
Problem with Getting Business Data from JMS 
The goal is to index the business message contents into Splunk 
Message Uncer...
Why use ZeroMQ 
Light Weight 
Multiple Client language support (Python, C++, Java, etc) 
Multiple design patterns (Pub/Sub...
Application Queue and ZeroMQ Example 
26 
Auto Load Balance 
1 
2
Example Python Sender 
context = zmq.Context() 
socket = context.socket(zmq.PUSH) 
socket.connect('tcp://127.0.0.1:5000') ...
Python Receiver (Scripted Input) 
context = zmq.Context() 
socket = context.socket(zmq.PULL) 
# Change address and port to...
Python Subscriber (Scripted Input) 
context = zmq.Context() 
socket = context.socket(zmq.SUB) 
socket.connect ("tcp://loca...
Parallel Pipeline Example 
30
Getting Events out of Splunk 
31 
Splunk SDK 
Use Cases: 
– In Depth processing of Splunk events in a queued manner 
– Use...
Java Example using SDK to load ZeroMQ 
String query=search; 
Job job = service.getJobs().create(query, queryArgs); 
while ...
Idle Computers at a Corporation 
33 
…
Idea: Use Ideas from SETI @ Home 
34
Idle Computers Put to Work Using JMS 
35 
…
Applications for Distributing Work 
Application Server would free up computing resources 
Work could be pushed to underuti...
Architecture 
Optional 
37
Algorithm 
Application servers push requests to queues, which may include data 
in the request object called a Unit of Wor...
Matrix Example High Level Architecture 
39
Search Language Against Matrix Result 
List Column Values of Each Stored Multiplied Matrix using Multikv 
40 
Screenshot h...
Search Language Against Matrix Result 
Visualize the Average for Columns 2 to 5 
41 
Screenshot here
Search Language Against Matrix Result 
Perform arbitrary math on aggregate columns 
42 
Screenshot here
Reference 
ZeroMQ 
– http://apps.splunk.com/app/1000/ 
– Blog: http://blogs.splunk.com/2012/06/08/zeromq-as-a-splunk-input...
Questions ?
THANK YOU 
ddallimore@splunk.com 
ndoshi@splunk.com
Upcoming SlideShare
Loading in …5
×

Splunk Conf 2014 - Getting the message

1,618 views

Published on

Presentation from Splunk Conf 14

Published in: Software
  • Be the first to comment

Splunk Conf 2014 - Getting the message

  1. 1. Copyright © 2014 Splunk Inc. Getting the Message Damien Dallimore Dev Evangelist , CSO Office @ Splunk Nimish Doshi Principal Systems Engineer @ Splunk
  2. 2. Disclaimer During the course of this presentation, we may make forward looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in the this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release. 2
  3. 3. Agenda 3 Damien’s Section What is messaging JMS + Demo AMQP + Demo Kafka + Demo Custom message handling Architecting for scale Nimish’s Section Using ZeroMQ Using JMS for underutilized computers Question time
  4. 4. Damien’s Section
  5. 5. 5 From Middle Earth Make Splunk Apps & Add-ons Messaging background
  6. 6. 6
  7. 7. apps.splunk.com github.com/damiendallimore 7
  8. 8. What is messaging ? Messaging infrastructures facilitate the sending/receiving of messages between distributed systems Message can be encoded in one of many available protocols A common paradigm involves producers and consumers exchanging via topics or queues 8 Topics (publish subscribe) Queues (point to point) TOPIC QUEUE
  9. 9. Why are messaging architectures used ? Integrating Legacy Systems Integrating Heterogeneous Systems Distributed Applications Cluster Communication High Performance Streaming 9
  10. 10. There’s a lot of information in the pipes 10
  11. 11. The data opportunity Easily tap into a massive source of valuable inflight data flowing around the veins Don’t need to access the application directly ,pull data off the messaging bus I can not think of a single industry vertical that does not use messaging 11
  12. 12. Getting this data into Splunk Many different messaging platforms and protocols JMS (Java Message Service) AMQP (Advanced Message Queueing Protocol) Kafka Nimish will cover some more uses cases also 12
  13. 13. JMS Not a messaging protocol , but a programming interface to many different underlying message providers WebsphereMQ , Tibco EMS , ActiveMQ , HornetQ , SonicMQ etc… Very prevalent in the enterprise software landscape DEMO 13
  14. 14. AMQP RabbitMQ Supports AMQP 0.9.1, 0.9, 0.8 Common in financial services and environments that need high performance and low latency DEMO 14
  15. 15. Kafka Cluster centric design = strong durability and fault tolerance Scales elastically Producers and Consumers communicate via topics in a Kafka node cluster Very popular with open source big data / streaming analytics solutions DEMO 15
  16. 16. Custom message handling These Modular Inputs can be used in a multitude of scenarios Message bodies can be anything : JSON, XML, CSV, Unstructured text, Binary Need to give the end user the ability to customize message processing So you can plugin your own custom handlers Need to write code , but it is really easy , and there are examples on GitHub I’m a big data pre processing fan 16
  17. 17. Cut the code 17
  18. 18. Compile, bundle into jar file, copy to Splunk 18
  19. 19. Declaratively apply it Let’s see if it works 19
  20. 20. Achieving desired scale AMQP Mod Input AMQP Queue 20 Single Splunk Instance With 1 Modular Input instance , only so much performance / throughput can be achieved You’ll hit limits with JVM heap , CPU , OS STDIN/STDOUT Buffer , Splunk indexing pipeline
  21. 21. So go Horizontal AMQP Queue 21 Splunk Indexer Cluster Universal Forwarders AMQP Broker AMQP Mod Input AMQP Mod Input
  22. 22. Nimish’s Section
  23. 23. About Me • Principal Systems Engineer at Splunk in the NorthEast • Session Speaker at all past Splunk .conf user conferences • Catch me on the Splunk Blogs 23
  24. 24. Problem with Getting Business Data from JMS The goal is to index the business message contents into Splunk Message Uncertainty Principal: If you de-queue the message to look at it, you have affected the TXN If you use various browse APIs for content, you may miss it – Message may have already been consumed by TXN Suggestion: Use a parallel queue to log the message – Suggestion: Try ZeroMQ 24
  25. 25. Why use ZeroMQ Light Weight Multiple Client language support (Python, C++, Java, etc) Multiple design patterns (Pub/Sub, Pipeline, Request/Reply, etc) Open Source with community support 25
  26. 26. Application Queue and ZeroMQ Example 26 Auto Load Balance 1 2
  27. 27. Example Python Sender context = zmq.Context() socket = context.socket(zmq.PUSH) socket.connect('tcp://127.0.0.1:5000') sleeptime=0.5 27 while True: num=random.randint(50,100) now = str(datetime.datetime.now()) sleep(sleeptime) payload = now + " Temperature=" + str(num) socket.send(payload)
  28. 28. Python Receiver (Scripted Input) context = zmq.Context() socket = context.socket(zmq.PULL) # Change address and port to match your environment socket.bind("tcp://127.0.0.1:5000") 28 while True: msg = socket.recv() print "%s" % msg except: print "exception"
  29. 29. Python Subscriber (Scripted Input) context = zmq.Context() socket = context.socket(zmq.SUB) socket.connect ("tcp://localhost:5556") # Subscribe to direction filter = "east" socket.setsockopt(zmq.SUBSCRIBE, filter) 29 while True: string = socket.recv() print string
  30. 30. Parallel Pipeline Example 30
  31. 31. Getting Events out of Splunk 31 Splunk SDK Use Cases: – In Depth processing of Splunk events in a queued manner – Use as pivot point to drop off events into a Complex Event Processor – Batch Processing of Splunk events outside of Splunk  Divide and Conquer Approach as seen in last slide
  32. 32. Java Example using SDK to load ZeroMQ String query=search; Job job = service.getJobs().create(query, queryArgs); while (!job.isDone()) { 32 Thread.sleep(100); job.refresh(); } // Get Query Results and store in String str… (Code Omitted) // Assuming single line events StringTokenizer st = new StringTokenizer(str, "n"); while(st.hasMoreTokens()) { String temp= st.nextToken(); sock.send(temp.getBytes(), 0); byte response[] = sock.recv(0); }
  33. 33. Idle Computers at a Corporation 33 …
  34. 34. Idea: Use Ideas from SETI @ Home 34
  35. 35. Idle Computers Put to Work Using JMS 35 …
  36. 36. Applications for Distributing Work Application Server would free up computing resources Work could be pushed to underutilized computers Examples: – Massive Mortgage Calculation Scenarios – Linear Optimization Problems – Matrix Multiplication – Compute all possible paths for combinatorics 36
  37. 37. Architecture Optional 37
  38. 38. Algorithm Application servers push requests to queues, which may include data in the request object called a Unit of Work JMS client implements doWork() interface to work with data Message Driven Bean receives finished work and implements doStore() interface What does this have to do with Splunk? – Time Series results can be stored in Splunk for further or historical analytics 38
  39. 39. Matrix Example High Level Architecture 39
  40. 40. Search Language Against Matrix Result List Column Values of Each Stored Multiplied Matrix using Multikv 40 Screenshot here
  41. 41. Search Language Against Matrix Result Visualize the Average for Columns 2 to 5 41 Screenshot here
  42. 42. Search Language Against Matrix Result Perform arbitrary math on aggregate columns 42 Screenshot here
  43. 43. Reference ZeroMQ – http://apps.splunk.com/app/1000/ – Blog: http://blogs.splunk.com/2012/06/08/zeromq-as-a-splunk-input/ Using JMS for Underutilized Computers – Github Reference: https://github.com/nimishdoshi/JMSClientApp/ – Blog: http://blogs.splunk.com/2014/04/11/splunk-as-a-recipient-on-the-jms-grid/ – Article:http://www.oracle.com/technetwork/articles/entarch/jms-distributed-work- 082249.html 43
  44. 44. Questions ?
  45. 45. THANK YOU ddallimore@splunk.com ndoshi@splunk.com

×