A Survey of the State-of-the-art
in Event Processing
Otávio M. de Carvalho, Eduardo Roloff, Philippe O. A. Navaux
Federal ...
Motivation
- We currently live in
a world of 2.8 trillion GB
- Data production rate
is growing 60% by year
- Every 60 seco...
Applications of Information Flow
Processing

3
The History Behind Event Processing Tools
●

●

●

The lastest advance in Event Processing can be characterized by two mai...
MapReduce is Still Important
●

●

The Google's research on MapReduce model was
fundamental to the evolution of event proc...
But MapReduce Model Isn't Fast Enought

x

6
State-of-the-art
in
Event Processing

7
Characterization of Tools
Name

Release Year

DBMS

DSMS

CEP

Distributed

Google Photon

2013

•

•

Walmart Muppet

201...
Characterization of Tools
Name

Release Year

DBMS

DSMS

CEP

Distributed

Google Photon

2013

•

•

Walmart Muppet

201...
Characterization of Tools
Name

Release Year

DBMS

DSMS

CEP

Distributed

Google Photon

2013

•

•

Walmart Muppet

201...
Characterization of Tools
Name

Release Year

DBMS

DSMS

CEP

Distributed

Google Photon

2013

•

•

Walmart Muppet

201...
Characterization of Tools
Name

Release Year

DBMS

DSMS

CEP

Distributed

Google Photon

2013

•

•

Walmart Muppet

201...
Characterization of Tools
Name

Release Year

DBMS

DSMS

CEP

Distributed

Google Photon

2013

•

•

Walmart Muppet

201...
Hadoop Influency
Name

Based on Hadoop and his core tools

Interact with Hadoop

Google Photon
Walmart Muppet
StreamDrill
...
Challenges in Distributed Event
Processing
●

How to program?

●

In what applications can it be used?

●

Now it scales, ...
Conclusions
●

●

●

MapReduce paper and later contributions have
collaborated to the changes into event
processing system...
Future Works
●

●

●

In our future works, we will select a subset of
these tools to analyze further
We will evaluate the ...
A Survey of the State-of-the-art in Event
Processing

Thanks!
Otávio M. de Carvalho, Eduardo Roloff, Philippe O. A. Navaux...
Upcoming SlideShare
Loading in …5
×

A Survey of the State-of-the-art in Event Processing

3,415 views

Published on

Survey Paper presentation about Event Processing tools current being developed, the history behind Event Processing tools and

Published in: Technology

A Survey of the State-of-the-art in Event Processing

  1. 1. A Survey of the State-of-the-art in Event Processing Otávio M. de Carvalho, Eduardo Roloff, Philippe O. A. Navaux Federal University of Rio Grande do Sul Parallel and Distributed Processing Group {omcarvalho,eroloff,navaux}@inf.ufrgs.br
  2. 2. Motivation - We currently live in a world of 2.8 trillion GB - Data production rate is growing 60% by year - Every 60 seconds, 600 new blog posts are published and 34,000 tweets are sent 2
  3. 3. Applications of Information Flow Processing 3
  4. 4. The History Behind Event Processing Tools ● ● ● The lastest advance in Event Processing can be characterized by two main sub-domains: CEPs and DSMSs systems that were developed in the early 90s Main problem: Today's systems present intersections between these two sub-domains A new naming convention was proposed: Information Flow Processing (IFP) ● Complex Event Processing Systems Those systems focuses on processing event notifications, with a special attention to their ordering relationships, to capture complex event patterns. ● Data Stream Management Systems Those systems focuses mainly on flowing data and the application of transformations over data 4
  5. 5. MapReduce is Still Important ● ● The Google's research on MapReduce model was fundamental to the evolution of event processing systems into distributed event processing systems The adoption of Apache Hadoop and his core tools has eased the process of developing distributed systems 5
  6. 6. But MapReduce Model Isn't Fast Enought x 6
  7. 7. State-of-the-art in Event Processing 7
  8. 8. Characterization of Tools Name Release Year DBMS DSMS CEP Distributed Google Photon 2013 • • Walmart Muppet 2012 • • StreamDrill 2012 SAP HANA 2011 Apache Storm 2011 Apache YARN 2011 Apache Flume 2011 • • Apache Kafka 2011 • • Apache S4 2011 • • Apache Chukwa 2010 • • HStreaming 2010 • • AMPLab Spark 2010 • • • VoltDB 2010 • • • Esper 2006 • StreamBase CEP 2003 • SQLstream 2003 • • • • • • • • • • • • • 8
  9. 9. Characterization of Tools Name Release Year DBMS DSMS CEP Distributed Google Photon 2013 • • Walmart Muppet 2012 • • StreamDrill 2012 SAP HANA 2011 Apache Storm 2011 Apache YARN 2011 Apache Flume 2011 • • Apache Kafka 2011 • • Apache S4 2011 • • Apache Chukwa 2010 • • HStreaming 2010 • • AMPLab Spark 2010 • • • VoltDB 2010 • • • Esper 2006 StreamBase CEP 2003 SQLstream 2003 • • • • • • • • Legacy Tools • • • • • • • 9
  10. 10. Characterization of Tools Name Release Year DBMS DSMS CEP Distributed Google Photon 2013 • • Walmart Muppet 2012 • • StreamDrill 2012 SAP HANA 2011 Apache Storm 2011 Apache YARN 2011 Apache Flume 2011 Apache Kafka 2011 Apache S4 • • In-memory databases • • • • • • • • • • • • 2011 • • Apache Chukwa 2010 • • HStreaming 2010 • • AMPLab Spark 2010 • • • VoltDB 2010 • • • Esper 2006 • StreamBase CEP 2003 • SQLstream 2003 • • • 10
  11. 11. Characterization of Tools Name Release Year DBMS DSMS CEP Distributed Google Photon 2013 • • Walmart Muppet 2012 • • StreamDrill 2012 SAP HANA 2011 Apache Storm 2011 Apache YARN 2011 Apache Flume 2011 Apache Kafka • • Flow processing • • • • • • • • • • 2011 • • Apache S4 2011 • • Apache Chukwa 2010 • • HStreaming 2010 • • AMPLab Spark 2010 • • • VoltDB 2010 • • • Esper 2006 • StreamBase CEP 2003 • SQLstream 2003 • • • 11
  12. 12. Characterization of Tools Name Release Year DBMS DSMS CEP Distributed Google Photon 2013 • • Walmart Muppet 2012 • • StreamDrill 2012 SAP HANA 2011 Apache Storm 2011 Apache YARN 2011 Apache Flume 2011 • • Apache Kafka 2011 • Apache S4 2011 General-purpose• systems • Apache Chukwa 2010 • • HStreaming 2010 • • AMPLab Spark 2010 • • • VoltDB 2010 • • • Esper 2006 • StreamBase CEP 2003 • SQLstream 2003 • • • • • • • • • • • • • • 12
  13. 13. Characterization of Tools Name Release Year DBMS DSMS CEP Distributed Google Photon 2013 • • Walmart Muppet 2012 • • StreamDrill 2012 SAP HANA 2011 Apache Storm 2011 Apache YARN 2011 Apache Flume 2011 • • Apache Kafka 2011 • • Apache S4 2011 • • Apache Chukwa 2010 • • HStreaming 2010 • • AMPLab Spark 2010 • • • VoltDB 2010 • • • Esper 2006 • StreamBase CEP 2003 • SQLstream 2003 • • • Specific to solve one • problem (top-k problem) • • • • • • • • • 13
  14. 14. Hadoop Influency Name Based on Hadoop and his core tools Interact with Hadoop Google Photon Walmart Muppet StreamDrill SAP HANA • Apache Storm Apache YARN • Apache Flume • Apache Kafka • Apache S4 Apache Chukwa • HStreaming • AMPLab Spark • VoltDB • Esper StreamBase CEP • SQLstream • 14
  15. 15. Challenges in Distributed Event Processing ● How to program? ● In what applications can it be used? ● Now it scales, but is the throughput the same? 15
  16. 16. Conclusions ● ● ● MapReduce paper and later contributions have collaborated to the changes into event processing systems Convergences between the tools still are causing naming problems It is not possible to determine if these implementations will converge into greater sets of tools of general-purpose systems or more specialized systems. 16
  17. 17. Future Works ● ● ● In our future works, we will select a subset of these tools to analyze further We will evaluate the performance of these selected tools And finally, we will apply one of them in a real HPC application 17
  18. 18. A Survey of the State-of-the-art in Event Processing Thanks! Otávio M. de Carvalho, Eduardo Roloff, Philippe O. A. Navaux Federal University of Rio Grande do Sul Parallel and Distributed Processing Group {omcarvalho,eroloff,navaux}@inf.ufrgs.br

×