This paper presents an analysis of the stream analytics market based on real world experience. The paper presents practical viewpoints of stream analytic platforms like Apache Storm, Spark Streaming, Apache Samza, AWS Kinesis, Salesforce Thunder and Azure Stream Analytics

1. A Practical Reference Guide to Stream
Analytics Solutions in the Enterprise

2. Contents
Overview.......................................................................................................................................................3
Key Characteristics of Modern Stream Analytics Solutions..........................................................................3
Challenges with Traditional Stream Analytic Platform Vendors...................................................................5
Stream Analytics in the Cloud.......................................................................................................................5
AWS Kinesis ..............................................................................................................................................6
Azure Stream Analytics ............................................................................................................................6
Salesforce Thunder...................................................................................................................................7
Stream Analytics On-Premise .......................................................................................................................8
Apache Storm...........................................................................................................................................8
Spark Streaming .......................................................................................................................................9
Apache Samza.........................................................................................................................................10
Summary.....................................................................................................................................................11

3. Overview
Stream analytics is becoming a foundational piece of real time analytics solutions.
The ability to process, analyze and react to data insights real time is an essential
component of modern enterprise solutions. While most enterprises clearly
understand the value proposition of stream analytics solutions, that hasn’t
translated onto technology adoptions. In our opinion, this technology adoption
challenge is a combination of the failure of traditional “complex event processing”
vendors to deliver modern stream analytic capabilities as well as the lack of
understanding about modern platforms that deliver a simpler and yet more
sophisticated stream analytics experience.
Having implemented a few stream analytic solutions for Global 2000 companies, we
decided to compile some of our experiences in this report. The goal is not to
produce an analyst report but rather to offer practical guidance to organizations
evaluating stream analytics solutions. The thoughts expressed in this report are the
result of practical experience implementations of stream analytic solutions and not
on specific product guidance or recommendation.
Key Characteristics of Modern Stream Analytics Solutions
While the principles of stream analytics solutions have been widely adopted in
specific domains like public markets, they are relatively novel to most industries. In
that sense, there is no well-established criteria of what makes an enterprise-ready
stream analytics solution. In our experience, there are a few factors that should be
considered.
Connectors to Line of Business Systems
Integration with existing line of business systems is an essential element of stream
analytics solutions. Often overlooked streaming data to and from enterprise
systems traditionally becomes one of the most complex and time consuming
aspects of the stream analytics implementations. To address that, enterprise-ready
stream analytic solutions should provide connectors to business systems that will

4. produce or consume data streams based on system events. Stream data connectors
are not only a huge time saver but enforce a very important consistency to how
data is being produced and consumed in stream analytics solutions.
Stream Replay and Simulation
Testing is one of the most difficult capabilities to implement in real world stream
analytics solution. To efficiently test stream analytics scenarios, a solution should
be able to replay specific data streams in the correct order. When evaluating
stream analytics platforms, organizations should consider solutions that enable the
recording and replaying of events in order to recreate real world conditions. This
capability will allow enterprises to recreate different real world scenarios by
replaying the correct streams of data.
Stream Tracking and Monitoring
Similar to the previous point, actively tracking and monitoring of events is an
essential element in enterprise-ready stream analytics solutions. This capability is a
foundational building block to enable the operational management, support and
compliance of stream analytics solutions. Beyond the infrastructure aspect of high
throughput event stream monitoring, enterprise-ready stream analytics solutions
should provide tools that allows dev.-ops and IT professionals to visualize,
understand and take action based on the runtime behavior of stream analytics
solutions.
Security and Data Privacy
Security and data privacy are key aspects of any enterprise data solution and
stream analytics is not the exception. Providing mechanisms to encrypt, sign and
protect data streams is required to address many of the security and compliance
requirements of mission critical enterprise solutions. In that sense, enterprises
should factor in the relevant security and data privacy capabilities when evaluating
stream analytic platforms.
Interfaces for Power Users

5. Stream analytics is typically considered an infrastructure capability. However, many
of the solutions powered by stream analytics platforms have a strong end user
component. As a result, power users often need to interact and extend aspects of
the solution in order to adapt to new scenarios. To enable this requirement,
enterprises should look for stream analytic solutions that provide robust tools for
power users in order to enable important self-service capabilities.
Integration with Existing Analytic Tools
The analytics and data visualization space has evolved drastically in the last few
years. Many new entrants in the market like Tableau or QlikView have become
common citizens in modern enterprises. The integration with mainstream analytics
tools is an important element of modern stream analytics solutions. This capability
allows business users to interact with complex stream analytics infrastructures
using a familiar toolset, which minimizes the friction for the adoption of these
solutions in the enterprise.
Challenges with Traditional Stream Analytic Platform Vendors
When evaluating stream analytic solutions, organizations tend to segment the
market on different groups. Based on our experience, grouping stream analytics
solutions based on their underlying topology (cloud or on-premise) is a very
effective way to start analyzing this market. While cloud and on-premise stream
analytics stack can be very similar on their functional capabilities, they drastically
differ in terms of their underlying infrastructure as well as the management and
scalability model.
The next section of this document will look at the most important stream analytics
platforms in both on-premise and cloud topologies.
Stream Analytics in the Cloud
Stream analytics have become an important component of platform as a service
(PaaS) solutions. The elastic scalability model of cloud infrastructure drastically
simplifies the implementation of stream analytics solutions. As a result, many of the
most sophisticated stream analytics platforms in the market are delivered as part of
PaaS offerings.

6. Below we detailed some of the most important players in the space.
AWS Kinesis
 Description: Amazon Kinesis is a platform for streaming data on AWS,
offering powerful services to make it easy to load and analyze streaming
data, and also providing the ability for you to build custom streaming data
applications for specialized needs. Web applications, mobile devices,
wearable’s, industrial sensors, and many software applications and services
can generate staggering amounts of streaming data – sometimes TBs per
hour – that need to be collected, stored, and processed continuously.
 Key Capabilities: AWS Kinesis is based on three fundamental building
blocks
o Kinesis Firehose: The Firehose provides an easy way to load
streaming data into AWS. By default, Kinesis Firehose can load data
streams into AWS services like redshift or S3. It can also batch,
compress, and encrypt the data before loading it, minimizing the
amount of storage used at the destination and increasing security.
o Kinesis Analytics: Kinesis Analytics executes SQL queries against
dynamic data streams produced by the Kinesis Firehose. Additionally,
Kinesis Analytics helps to model actions as a result of stream queries.
o Kinesis Streams: Kinesis Streams provides a model to implement
data streams that can be processed by Kinesis applications. This
component of the platform provides the programming models to build
connectors that generate data streams from various data systems
including the services in the AWS platform.
 Challenges: AWS Kinesis is one of the most complete stream analytics
platforms in the market. However, like any new solution it presents well-
known challenges when implemented in real world solutions. The lack of tools
for non-developers as well as the limited simulation and even tracking
capabilities are some of the most notorious limitations of AWS Kinesis.
Azure Stream Analytics
 Description: Azure Stream Analytics (ASA) is a fully managed, real-time
event-processing engine that helps to unlock deep insights from data.

7. Stream Analytics makes it easy to set up real-time analytic computations on
data streaming from devices, sensors, web sites, social media, applications,
infrastructure systems, and more. Using the Azure portal, you can author a
Stream Analytics job specifying the input source of the streaming data, the
output sink for the results of your job, and a data transformation expressed
in an SQL-like language. You can monitor and adjust the scale/speed of jobs
in the Azure portal to scale from a few kilobytes to a gigabyte or more of
events processed per second.
 Key Capabilities: Azure Stream Analytics provides some key capabilities
that should be considered during its evaluation.
o IOT Hub and Event Hubs: Azure Stream Analytics integrates with
Event Hubs and IOT Hub to process large volume of events per second
and scale linearly.
o SQL Queries: Azure Stream Analytics powers the authoring and
execution of SQL like queries over dynamic streams of data.
o Support for Storm and Spark Streaming: In addition to its native
stream analytics platform, Azure provides support for Apache Storm
and Spark Streaming. As a result, solutions built on the Azure platform
can take advantage of various stream analytic model depending on the
requirements.
 Challenges: Similar to AWS Kinesis, Azure Stream Analytics lacks strong
support and tooling for non-developers. Additionally, the platform hasn’t yet
developed an ecosystem of connectors that produce and consume streams of
data from Azure Stream Analytics.
Salesforce.com Thunder
 Description: Based on Apache Storm, Salesforce.com Thunder is a very
scalable event-processing engine, designed to ingest and orchestrate billions
of events from the connected world in real time. Thunder uses smart
technology to reveal insights that were invisible and allow anyone to take
proactive personalized actions from any device.
 Key Capabilities: Despite being a new entrant in the market,
Salesforce.com Thunder includes some key capabilities such as the following:

8. o Orchestration: Salesforce.com Thunder provides a end user friendly
model to author and manage rules that model actions based on the
results extracted from data streams.
o SQL Queries: Salesforce.com Thunder powers the authoring and
execution of SQL like queries over dynamic streams of data.
o Integrated with Salesforce.com Cloud Products: Without any
required customization, Salesforce.com Thunder integrates with
different Salesforce.com Cloud products, which facilitates its
immediate applicability in business scenarios.
 Challenges: Compared to platforms like AWS and Azure, Salesforce.com
Thunder provides a more limited solution from the infrastructure standpoint.
Additionally, Salesforce.com Thunder still lacks connectors, stream producers
and even integration with third party services outside the Salesforce.com
platform
Stream Analytics On-Premise
While cloud stream analytics are incredibly advanced from the capability standpoint,
they are prohibited for many organizations subjected to regulatory and compliance
requirements. In those scenarios, on-premise stream analytics solutions represent
a powerful alternative. One important distinction from the platforms in the market,
we believe Apache Storm, Spark Streaming and Apache Samza represent the most
advanced technology stacks.
Apache Storm
 Description: Storm is a distributed real-time computation system for
processing large volumes of high-velocity data. Storm is extremely fast, with
the ability to process over a million records per second per node on a cluster
of modest size. Enterprises harness this speed and combine it with other data
access applications in Apache Hadoop to prevent undesirable events or to
optimize positive outcomes.
 Key Capabilities:

9. o Modular Architecture: Storm offers a very modular architecture in
which components can be composed onto really complex stream
analytics solutions.
o Trident: Trident offers a simpler interface to author applications on
top of Storm. This model facilitates the implementation of stream
analytics applications without having to be completely familiar with
Storm’s architecture.
o Connector Framework: Storm provides a number of connectors
(Spouts and Bolts) to different data platforms and provides models
that allow developers to implement custom connectors to new
systems.
 Challenges: The biggest challenge with enterprise implementations of Storm
is the lack of sophisticated, enterprise-ready tools that are well known and
adopted within IT environments. However, the Storm community and
implementers have slowly started to make progress to address some of this
limitations.
Spark Streaming
 Description: Spark is rapidly becoming one of the most important big data
platforms in modern solutions. Spark Streaming is an extension of the core
Spark API that enables scalable, high-throughput, fault-tolerant stream
processing of live data streams. Data can be ingested from many sources like
Kafka, Flume, Twitter, ZeroMQ, Kinesis, or TCP sockets, and can be
processed using complex algorithms expressed with high-level functions like
map, reduce, join and window. Finally, processed data can be pushed out to
file systems, databases, and live dashboards.
 Key Capabilities:
o Integration with Spark: Spark streaming is tightly integrated with
the other components of the Spark platform including Spark MLib or
Spark SQL. This feature facilitates the implementation of feature-rich
stream analytics solutions without abandoning the Spark platform.

10. o Simplicity: Spark Streaming provides a super simple programming
model that allow developers author complex stream analytic solutions
without having to become an expert in the space.
o Big Data Platform Ecosystem: Spark is being widely adopted by
most big data platform vendors such as Cloudera, Hortonworks or
MapR. This ecosystem is indirectly influencing the adoption of Spark
Streaming as the default engine to implement stream analytics
solutions in big data platforms.
Challenges: The dependencies on Spark bring notable advantages to Spark
Streaming but also introduce very notable challenges. Extending stream analytic
solutions beyond the capabilities provided by the Spark can result in very complex
implementations. Another interesting challenge of Spark Streaming solutions is the
integration with third party systems. Based on the MPP requirements of Spark,
integrating third party systems into Spark requires a considerable engineering
effort compared to some of the other alternatives in the market.
Apache Samza
 Description: Apache Samza is a distributed stream-processing framework. It
was initially created in LinkedIn as an alternative to platforms like Apache Storm.
Samza uses Apache Kafka for messaging, and Apache Hadoop YARN to provide
fault tolerance, processor isolation, security, and resource management.
Although a recent entrant in the stream analytics space, the current feature set is
quite impressive.
 Key Capabilities:
o Extremely Complete Stream Processing Engine: Apache Samza
was designed to abstract some of the most complex aspects of stream
processing from the application developers. By default, Samza includes
capabilities such as state management, order processing, data
buffering etc. which are incredibly hard to implement in stream
analytic solutions.
o Simplicity: Apache Samza provides a super simple programming
model that allows developers author complex stream analytic

11. solutions. Additionally, Samza was authored to support multi-language
applications even though the initial implementation has been
constrained to JVM languages.
o Integration with Apache Kafka: Samza relies Apache Kafka for its
message passing models. Also built by LinkedIn, Kafka is quickly
becoming one of the most robust messaging platforms in the market
and is experiencing relevant adoption in the enterprise. The current
model makes Apache Samza a great choice for organizations looking
to expand their Kafka infrastructure with stream processing
capabilities.
Challenges: Apache Samza is the newest entrant in the stream analytics space. In
that sense, Samza still has limitations in terms of the integration with third party
systems, the robustness of its management tool stack and other capabilities
required to guarantee a mainstream adoption in the enterprise. Additionally, the
Samza developer and customer communities are relatively small compared to other
platforms.
Summary
Stream analytics is quickly becoming one of the most important elements of
modern enterprise business intelligence solutions. When evaluating stream
analytics solutions, organizations should look for capabilities such as integration
with third party systems, tracking and management tools etc. that will facilitate the
adoption in an enterprise environment. This paper provided an overview of the key
capabilities that are relevant to implement stream analytic solution in the
enterprise.
The stream analytic ecosystem can be divided between cloud and on-premise
platforms. In the cloud space, platforms like Azure, AWS or Salesforce.com have
released some of the most innovative stream analytic solutions in the market. Many
of those solutions are based on stream processing platforms like Apache Storm,
Spark Streaming or Apache Samza, which can be adopted on premise. This paper
included an analysis of some of the key stream analytic platforms including
strengths and weaknesses based on our experience in real world implementations.