PowerPoint from the Nebraska and Iowa Splunk User Group by Al Liebl. Collect, process and distribute data to Splunk and other destinations in milliseconds with real-time stream processing. Continuously collect high-velocity, high-volume data from diverse sources and distribute insights to multiple destinations in milliseconds

1. © 2019 SPLUNK INC.
Data Stream
Processor
Collect, Process, then Deliver Data in milliseconds

2. During the course of this presentation, we may make forward-looking statements
regarding future events or the expected performance of the company. We caution
you that such statements reflect our current expectations and estimates based on
factors currently known to us and that actual events or results could differ
materially. For important factors that may cause actual results to differ from those
contained in our forward-looking statements, please review our filings with the SEC.
The forward-looking statements made in this presentation are being made as of the
time and date of its live presentation. If reviewed after its live presentation, this
presentation may not contain current or accurate information. We do not assume
any obligation to update any forward-looking statements we may make. In addition,
any information about our roadmap outlines our general product direction and is
subject to change at any time without notice. It is for informational purposes only
and shall not be incorporated into any contract or other commitment. Splunk
undertakes no obligation either to develop the features or functionality described or
to include any such feature or functionality in a future release.
Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud,
Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in
the United States and other countries. All other brand names, product names, or
trademarks belong to their respective owners. © 2019 Splunk Inc. All rights
reserved.
Forward-
Looking
Statements
© 2019 SPLUNK INC.

3. © 2019 SPLUNK INC.
Our World
Never Stops
Evolving.
New Ideas. New Devices. New Processes.
© 2019 SPLUNK INC.

4. © 2019 SPLUNK INC.
Turning
Real-time
Data Into
Action
is Hard
Multiple
Users
Increasing Data
Volumes
Data Storage
& Privacy
Regulations
Numerous
Data Sources
Various
Systems
Increasing
Data
Velocity
© 2019 SPLUNK INC.

5. © 2019 SPLUNK INC.
Organizations need a solution to guarantee
efficient data delivery across the enterprise
Real-time Data
Transformation
Continuous, Real-
time Insights
Enterprise-Wide
Data Delivery
Format and remove
unnecessary or noisy data
as events occur
Detect specific conditions
or data patterns before
data reaches its
destination
Guarantee delivery of high-
volume,high-velocity data
to multiple systems

6. © 2019 SPLUNK INC.
Splunk Data Stream Processor
Log Files
Online
Shopping Cart
Cell Phones
and Devices
RFID
Messaging
Patient
Generated
Data
Servers
Web Services
Call Detail
Records
Protect sensitive data
Take action on data in
motion
Turn raw data into high-
value information
Distribute data to
Splunk or other
destinations
Filter
Format
Enrich
Mask Sensitive Data
Detect data patterns or conditions
Aggregate
Normalize Transform
Track and monitor pipeline health
Splunk Data Stream Processor
A real-time stream processing solution that collects, processes and delivers data to
Splunk and other destinations in milliseconds
Data Warehouse
Public Cloud
Message Bus

7. © 2019 SPLUNK INC.
Use Cases
Filter or route noisy
data to specific
destinations
Data Routing
Filtering/
Noise
Removal
Data
Formatting
Guarantee delivery of
high-volume, high-
velocity data to multiple
destinations
Format or organize data
using various functions
based on specified
conditions
Aggregate data based on
specific conditions and
identify abnormal patterns
in data
Data
Aggregation
DATA IN MOTION

8. © 2019 SPLUNK INC.
▶ Represents the operations to read, transform, and write data
▶ All data flows through a pipeline
Data pipeline

9. © 2019 SPLUNK INC.
▶ Read or receive data
• Use connections to connect to data sources
Source function

10. © 2019 SPLUNK INC.
▶ Connector
• Know how to connect to external systems
• Can be push-based or pull-based
• Can be services that passively receive
data
▶ Connection
• A configured instance of a connector
• Where to connect? (i.e. list of host names,
S3 bucket name, DB connection URL)
• How to authenticate? (i.e.
username/password, AWS access keys,
client certificates)
Connector and
connection

11. © 2019 SPLUNK INC.
▶ Connector services
• Provide APIs for clients to send data to
• Ingest REST API
• Forwarders service
• Write the data to the Splunk Firehose
▶ Push-based connectors
•Connect to an external message bus
• Apache Kafka, Azure EventHubs, AWS Kinesis, etc.
• Data bypasses the Splunk Firehose
▶Pull-based connectors
• Are scheduled to pull new data from an external source
•Each time they run, they connect and fetch data
• Connectors then write data to the Splunk Firehose
Types of connectors

12. © 2019 SPLUNK INC.
▶ Filters, transforms, or aggregates records
• Operates on each record of a stream
• Can be stateful or stateless
• Can be configured with advanced logic
Streaming function

13. © 2019 SPLUNK INC.
▶ Writes data to a destination
Sink function

14. © 2019 SPLUNK INC.
▶ A partially configured pipeline that can be used to create a new pipeline
▶ When you create a pipeline from a template, it pre-populates the graph
Template

15. © 2019 SPLUNK INC.
DSP Demo

16. © 2019 SPLUNK INC.
Overall system
UI Controls
AST
Generator
DSP Frontend
Flink Cluster
DSP
Configuration
Service
REST API
Compiler
Kafka
Indexer
Data
Producers
Kubernetes

17. © 2019 SPLUNK INC.
DSP Architecture
Splunk Solution
Universal
Forwarder
Heavy
Forwarder
Sources
S2S Service
HTTP Service
Pull-based
Connectors
Collect
Service
Messaging
Service
Streaming Service
Stream
Processing
Delivery
Service
Splunk
Instance
Splunk
Instance
Kafka,
Kinesis
Data Pipelines
Data Stream Processor
Splunk Solution

18. © 2019 SPLUNK INC.
Data producers
REST API
Forwarders
Service
Apache Flink
REST Client
Forwarder
Remote Data
Source
External
Message
Bus
LSDC
Kafka
Message
Bus
Client

19. © 2019 SPLUNK INC.
Technical Requirements
Hardware Requirements
● Minimum Node Requirement
○ CPU: 8 core (16 recommended)
○ Memory: 64GB (128GB
recommended)
○ Network: 10Gbps
○ Storage: 1TB
● Minimum 5 Node Cluster
Get Started Today!
Supported Data Sources*: Apache
Kafka ®, Amazon Kinesis, Amazon
S3, CloudTrail, Event Hubs, REST
APIs, Splunk (Universal Forwarder,
Heavy Weight Forwarder)
Supported Destinations*: Apache
Kafka ®, Amazon Kinesis, Splunk
Infrastructure Based Pricing
(vCPUs)
*More sources and destinations to
come in future releases

20. Thank You
© 2019 SPLUNK INC.