SlideShare a Scribd company logo
1 of 18
Download to read offline
A Survey of the State-of-the-art
in Event Processing
Otávio M. de Carvalho, Eduardo Roloff, Philippe O. A. Navaux
Federal University of Rio Grande do Sul
Parallel and Distributed Processing Group
{omcarvalho,eroloff,navaux}@inf.ufrgs.br
Motivation
- We currently live in
a world of 2.8 trillion GB
- Data production rate
is growing 60% by year
- Every 60 seconds, 600 new blog posts are published
and 34,000 tweets are sent

2
Applications of Information Flow
Processing

3
The History Behind Event Processing Tools
●

●

●

The lastest advance in Event Processing can be characterized by two main
sub-domains: CEPs and DSMSs systems that were developed in the early 90s
Main problem: Today's systems present intersections between these two
sub-domains
A new naming convention was proposed: Information Flow Processing (IFP)

●

Complex Event Processing
Systems
Those systems focuses on
processing event
notifications, with a special
attention to their ordering
relationships, to capture
complex event patterns.

●

Data Stream Management
Systems
Those systems focuses
mainly on flowing data and
the application of
transformations over data

4
MapReduce is Still Important
●

●

The Google's research on MapReduce model was
fundamental to the evolution of event processing systems into
distributed event processing systems
The adoption of Apache Hadoop and his core tools has eased
the process of developing distributed systems

5
But MapReduce Model Isn't Fast Enought

x

6
State-of-the-art
in
Event Processing

7
Characterization of Tools
Name

Release Year

DBMS

DSMS

CEP

Distributed

Google Photon

2013

•

•

Walmart Muppet

2012

•

•

StreamDrill

2012

SAP HANA

2011

Apache Storm

2011

Apache YARN

2011

Apache Flume

2011

•

•

Apache Kafka

2011

•

•

Apache S4

2011

•

•

Apache Chukwa

2010

•

•

HStreaming

2010

•

•

AMPLab Spark

2010

•

•

•

VoltDB

2010

•

•

•

Esper

2006

•

StreamBase CEP

2003

•

SQLstream

2003

•
•

•

•
•

•

•

•

•

•

•

•
•

8
Characterization of Tools
Name

Release Year

DBMS

DSMS

CEP

Distributed

Google Photon

2013

•

•

Walmart Muppet

2012

•

•

StreamDrill

2012

SAP HANA

2011

Apache Storm

2011

Apache YARN

2011

Apache Flume

2011

•

•

Apache Kafka

2011

•

•

Apache S4

2011

•

•

Apache Chukwa

2010

•

•

HStreaming

2010

•

•

AMPLab Spark

2010

•

•

•

VoltDB

2010

•

•

•

Esper

2006

StreamBase CEP

2003

SQLstream

2003

•
•

•

•
•

•

•

•

Legacy Tools

•

•

•
•
•

•
•

9
Characterization of Tools
Name

Release Year

DBMS

DSMS

CEP

Distributed

Google Photon

2013

•

•

Walmart Muppet

2012

•

•

StreamDrill

2012

SAP HANA

2011

Apache Storm

2011

Apache YARN

2011

Apache Flume

2011

Apache Kafka

2011

Apache S4

•
•

In-memory
databases

•

•
•

•

•

•

•

•

•

•

•

•

2011

•

•

Apache Chukwa

2010

•

•

HStreaming

2010

•

•

AMPLab Spark

2010

•

•

•

VoltDB

2010

•

•

•

Esper

2006

•

StreamBase CEP

2003

•

SQLstream

2003

•

•
•

10
Characterization of Tools
Name

Release Year

DBMS

DSMS

CEP

Distributed

Google Photon

2013

•

•

Walmart Muppet

2012

•

•

StreamDrill

2012

SAP HANA

2011

Apache Storm

2011

Apache YARN

2011

Apache Flume

2011

Apache Kafka

•
•

Flow processing

•

•
•

•

•

•

•

•

•

•

2011

•

•

Apache S4

2011

•

•

Apache Chukwa

2010

•

•

HStreaming

2010

•

•

AMPLab Spark

2010

•

•

•

VoltDB

2010

•

•

•

Esper

2006

•

StreamBase CEP

2003

•

SQLstream

2003

•

•
•

11
Characterization of Tools
Name

Release Year

DBMS

DSMS

CEP

Distributed

Google Photon

2013

•

•

Walmart Muppet

2012

•

•

StreamDrill

2012

SAP HANA

2011

Apache Storm

2011

Apache YARN

2011

Apache Flume

2011

•

•

Apache Kafka

2011

•

Apache S4

2011

General-purpose•
systems
•

Apache Chukwa

2010

•

•

HStreaming

2010

•

•

AMPLab Spark

2010

•

•

•

VoltDB

2010

•

•

•

Esper

2006

•

StreamBase CEP

2003

•

SQLstream

2003

•
•

•

•
•

•

•

•

•

•

•

•

•
•

12
Characterization of Tools
Name

Release Year

DBMS

DSMS

CEP

Distributed

Google Photon

2013

•

•

Walmart Muppet

2012

•

•

StreamDrill

2012

SAP HANA

2011

Apache Storm

2011

Apache YARN

2011

Apache Flume

2011

•

•

Apache Kafka

2011

•

•

Apache S4

2011

•

•

Apache Chukwa

2010

•

•

HStreaming

2010

•

•

AMPLab Spark

2010

•

•

•

VoltDB

2010

•

•

•

Esper

2006

•

StreamBase CEP

2003

•

SQLstream

2003

•
•

•

Specific to solve one
•
problem (top-k problem)
•

•

•

•
•
•

•

•
•

13
Hadoop Influency
Name

Based on Hadoop and his core tools

Interact with Hadoop

Google Photon
Walmart Muppet
StreamDrill
SAP HANA

•

Apache Storm
Apache YARN

•

Apache Flume

•

Apache Kafka

•

Apache S4
Apache Chukwa

•

HStreaming

•

AMPLab Spark

•

VoltDB

•

Esper
StreamBase CEP

•

SQLstream

•

14
Challenges in Distributed Event
Processing
●

How to program?

●

In what applications can it be used?

●

Now it scales, but is the throughput the
same?

15
Conclusions
●

●

●

MapReduce paper and later contributions have
collaborated to the changes into event
processing systems
Convergences between the tools still are
causing naming problems
It is not possible to determine if these
implementations will converge into greater sets
of tools of general-purpose systems or more
specialized systems.

16
Future Works
●

●

●

In our future works, we will select a subset of
these tools to analyze further
We will evaluate the performance of these
selected tools
And finally, we will apply one of them in a real
HPC application

17
A Survey of the State-of-the-art in Event
Processing

Thanks!
Otávio M. de Carvalho, Eduardo Roloff, Philippe O. A. Navaux
Federal University of Rio Grande do Sul
Parallel and Distributed Processing Group
{omcarvalho,eroloff,navaux}@inf.ufrgs.br

More Related Content

Similar to A Survey of the State-of-the-art in Event Processing

Searching The Enterprise Data Lake With Solr - Watch Us Do It!: Presented by...
Searching The Enterprise Data Lake With Solr  - Watch Us Do It!: Presented by...Searching The Enterprise Data Lake With Solr  - Watch Us Do It!: Presented by...
Searching The Enterprise Data Lake With Solr - Watch Us Do It!: Presented by...Lucidworks
 
Azure Cafe Marketplace with Hortonworks March 31 2016
Azure Cafe Marketplace with Hortonworks March 31 2016Azure Cafe Marketplace with Hortonworks March 31 2016
Azure Cafe Marketplace with Hortonworks March 31 2016Joan Novino
 
Stinger.Next by Alan Gates of Hortonworks
Stinger.Next by Alan Gates of HortonworksStinger.Next by Alan Gates of Hortonworks
Stinger.Next by Alan Gates of HortonworksData Con LA
 
Spark Summit EU talk by Stephan Kessler
Spark Summit EU talk by Stephan KesslerSpark Summit EU talk by Stephan Kessler
Spark Summit EU talk by Stephan KesslerSpark Summit
 
Configurando Aplicaciones para Réplicas de Lectura de SQL-Server AlwaysOn - C...
Configurando Aplicaciones para Réplicas de Lectura de SQL-Server AlwaysOn - C...Configurando Aplicaciones para Réplicas de Lectura de SQL-Server AlwaysOn - C...
Configurando Aplicaciones para Réplicas de Lectura de SQL-Server AlwaysOn - C...SpanishPASSVC
 
Architecting the Future of Big Data and Search
Architecting the Future of Big Data and SearchArchitecting the Future of Big Data and Search
Architecting the Future of Big Data and SearchHortonworks
 
Scalable Preservation Workflows
Scalable Preservation WorkflowsScalable Preservation Workflows
Scalable Preservation WorkflowsSCAPE Project
 
Bootcamp Data Science using Cloudera
Bootcamp Data Science using ClouderaBootcamp Data Science using Cloudera
Bootcamp Data Science using ClouderaAntónio Rodrigues
 
SAP HANA Migration Deck.pptx
SAP HANA Migration Deck.pptxSAP HANA Migration Deck.pptx
SAP HANA Migration Deck.pptxSingbBablu
 
Bring Your SAP and Enterprise Data to Hadoop, Kafka, and the Cloud
Bring Your SAP and Enterprise Data to Hadoop, Kafka, and the CloudBring Your SAP and Enterprise Data to Hadoop, Kafka, and the Cloud
Bring Your SAP and Enterprise Data to Hadoop, Kafka, and the CloudDataWorks Summit
 
Architecting Big Data Ingest & Manipulation
Architecting Big Data Ingest & ManipulationArchitecting Big Data Ingest & Manipulation
Architecting Big Data Ingest & ManipulationGeorge Long
 
Enabling the Real Time Analytical Enterprise
Enabling the Real Time Analytical EnterpriseEnabling the Real Time Analytical Enterprise
Enabling the Real Time Analytical EnterpriseHortonworks
 
Using semantic web technologies for exploratory olap a survey
Using semantic web technologies for exploratory olap a surveyUsing semantic web technologies for exploratory olap a survey
Using semantic web technologies for exploratory olap a surveyieeepondy
 
Elastic Data Analytics Platform @Datadog
Elastic Data Analytics Platform @DatadogElastic Data Analytics Platform @Datadog
Elastic Data Analytics Platform @DatadogC4Media
 
Frances McNamara - Kuali OLE Implementation at University of Chicago
Frances McNamara - Kuali OLE Implementation at University of ChicagoFrances McNamara - Kuali OLE Implementation at University of Chicago
Frances McNamara - Kuali OLE Implementation at University of ChicagoKuali Days UK
 
How city of chicago boosts their sap business objects environment prepares fo...
How city of chicago boosts their sap business objects environment prepares fo...How city of chicago boosts their sap business objects environment prepares fo...
How city of chicago boosts their sap business objects environment prepares fo...Sebastien Goiffon
 
2017 big data landscape and cutting edge innovations public
2017 big data landscape and cutting edge innovations public2017 big data landscape and cutting edge innovations public
2017 big data landscape and cutting edge innovations publicEvans Ye
 

Similar to A Survey of the State-of-the-art in Event Processing (20)

Searching The Enterprise Data Lake With Solr - Watch Us Do It!: Presented by...
Searching The Enterprise Data Lake With Solr  - Watch Us Do It!: Presented by...Searching The Enterprise Data Lake With Solr  - Watch Us Do It!: Presented by...
Searching The Enterprise Data Lake With Solr - Watch Us Do It!: Presented by...
 
Azure Cafe Marketplace with Hortonworks March 31 2016
Azure Cafe Marketplace with Hortonworks March 31 2016Azure Cafe Marketplace with Hortonworks March 31 2016
Azure Cafe Marketplace with Hortonworks March 31 2016
 
Stinger.Next by Alan Gates of Hortonworks
Stinger.Next by Alan Gates of HortonworksStinger.Next by Alan Gates of Hortonworks
Stinger.Next by Alan Gates of Hortonworks
 
Spark Summit EU talk by Stephan Kessler
Spark Summit EU talk by Stephan KesslerSpark Summit EU talk by Stephan Kessler
Spark Summit EU talk by Stephan Kessler
 
Configurando Aplicaciones para Réplicas de Lectura de SQL-Server AlwaysOn - C...
Configurando Aplicaciones para Réplicas de Lectura de SQL-Server AlwaysOn - C...Configurando Aplicaciones para Réplicas de Lectura de SQL-Server AlwaysOn - C...
Configurando Aplicaciones para Réplicas de Lectura de SQL-Server AlwaysOn - C...
 
Architecting the Future of Big Data and Search
Architecting the Future of Big Data and SearchArchitecting the Future of Big Data and Search
Architecting the Future of Big Data and Search
 
Scalable Preservation Workflows
Scalable Preservation WorkflowsScalable Preservation Workflows
Scalable Preservation Workflows
 
Bootcamp Data Science using Cloudera
Bootcamp Data Science using ClouderaBootcamp Data Science using Cloudera
Bootcamp Data Science using Cloudera
 
SAP HANA Migration Deck.pptx
SAP HANA Migration Deck.pptxSAP HANA Migration Deck.pptx
SAP HANA Migration Deck.pptx
 
Bring Your SAP and Enterprise Data to Hadoop, Kafka, and the Cloud
Bring Your SAP and Enterprise Data to Hadoop, Kafka, and the CloudBring Your SAP and Enterprise Data to Hadoop, Kafka, and the Cloud
Bring Your SAP and Enterprise Data to Hadoop, Kafka, and the Cloud
 
Architecting Big Data Ingest & Manipulation
Architecting Big Data Ingest & ManipulationArchitecting Big Data Ingest & Manipulation
Architecting Big Data Ingest & Manipulation
 
Enabling the Real Time Analytical Enterprise
Enabling the Real Time Analytical EnterpriseEnabling the Real Time Analytical Enterprise
Enabling the Real Time Analytical Enterprise
 
Dev Ops Training
Dev Ops TrainingDev Ops Training
Dev Ops Training
 
Using semantic web technologies for exploratory olap a survey
Using semantic web technologies for exploratory olap a surveyUsing semantic web technologies for exploratory olap a survey
Using semantic web technologies for exploratory olap a survey
 
NISO/DCMI September 25 Webinar: Implementing Linked Data in Developing Countr...
NISO/DCMI September 25 Webinar: Implementing Linked Data in Developing Countr...NISO/DCMI September 25 Webinar: Implementing Linked Data in Developing Countr...
NISO/DCMI September 25 Webinar: Implementing Linked Data in Developing Countr...
 
Elastic Data Analytics Platform @Datadog
Elastic Data Analytics Platform @DatadogElastic Data Analytics Platform @Datadog
Elastic Data Analytics Platform @Datadog
 
Frances McNamara - Kuali OLE Implementation at University of Chicago
Frances McNamara - Kuali OLE Implementation at University of ChicagoFrances McNamara - Kuali OLE Implementation at University of Chicago
Frances McNamara - Kuali OLE Implementation at University of Chicago
 
What's new in Spark 2.0?
What's new in Spark 2.0?What's new in Spark 2.0?
What's new in Spark 2.0?
 
How city of chicago boosts their sap business objects environment prepares fo...
How city of chicago boosts their sap business objects environment prepares fo...How city of chicago boosts their sap business objects environment prepares fo...
How city of chicago boosts their sap business objects environment prepares fo...
 
2017 big data landscape and cutting edge innovations public
2017 big data landscape and cutting edge innovations public2017 big data landscape and cutting edge innovations public
2017 big data landscape and cutting edge innovations public
 

More from Otávio Carvalho

Non-Kafkaesque Apache Kafka - Yottabyte 2018
Non-Kafkaesque Apache Kafka - Yottabyte 2018Non-Kafkaesque Apache Kafka - Yottabyte 2018
Non-Kafkaesque Apache Kafka - Yottabyte 2018Otávio Carvalho
 
GaruaGeo: Global Scale Data Aggregation in Hybrid Edge and Cloud Computing En...
GaruaGeo: Global Scale Data Aggregation in Hybrid Edge and Cloud Computing En...GaruaGeo: Global Scale Data Aggregation in Hybrid Edge and Cloud Computing En...
GaruaGeo: Global Scale Data Aggregation in Hybrid Edge and Cloud Computing En...Otávio Carvalho
 
IoT Workload Distribution Impact Between Edge and Cloud Computing in a Smart ...
IoT Workload Distribution Impact Between Edge and Cloud Computing in a Smart ...IoT Workload Distribution Impact Between Edge and Cloud Computing in a Smart ...
IoT Workload Distribution Impact Between Edge and Cloud Computing in a Smart ...Otávio Carvalho
 
Stream Processing - ThoughtWorks Architecture Group - 2017
Stream Processing - ThoughtWorks Architecture Group - 2017Stream Processing - ThoughtWorks Architecture Group - 2017
Stream Processing - ThoughtWorks Architecture Group - 2017Otávio Carvalho
 
Stream Processing: Uma visão geral - TDC Porto Alegre / FISL 17
Stream Processing: Uma visão geral - TDC Porto Alegre / FISL 17Stream Processing: Uma visão geral - TDC Porto Alegre / FISL 17
Stream Processing: Uma visão geral - TDC Porto Alegre / FISL 17Otávio Carvalho
 
Apache Kafka - Free Friday
Apache Kafka - Free FridayApache Kafka - Free Friday
Apache Kafka - Free FridayOtávio Carvalho
 
Distributed Near Real-Time Processing of Sensor Network Data Flows for Smart ...
Distributed Near Real-Time Processing of Sensor Network Data Flows for Smart ...Distributed Near Real-Time Processing of Sensor Network Data Flows for Smart ...
Distributed Near Real-Time Processing of Sensor Network Data Flows for Smart ...Otávio Carvalho
 
Análise e Caracterização das Novas Ferramentas para Computação em Nuvem
Análise e Caracterização das Novas Ferramentas para Computação em NuvemAnálise e Caracterização das Novas Ferramentas para Computação em Nuvem
Análise e Caracterização das Novas Ferramentas para Computação em NuvemOtávio Carvalho
 
Utilização de traços de execução para migração de aplicações para a nuvem
Utilização de traços de execução para migração de aplicações para a nuvemUtilização de traços de execução para migração de aplicações para a nuvem
Utilização de traços de execução para migração de aplicações para a nuvemOtávio Carvalho
 

More from Otávio Carvalho (9)

Non-Kafkaesque Apache Kafka - Yottabyte 2018
Non-Kafkaesque Apache Kafka - Yottabyte 2018Non-Kafkaesque Apache Kafka - Yottabyte 2018
Non-Kafkaesque Apache Kafka - Yottabyte 2018
 
GaruaGeo: Global Scale Data Aggregation in Hybrid Edge and Cloud Computing En...
GaruaGeo: Global Scale Data Aggregation in Hybrid Edge and Cloud Computing En...GaruaGeo: Global Scale Data Aggregation in Hybrid Edge and Cloud Computing En...
GaruaGeo: Global Scale Data Aggregation in Hybrid Edge and Cloud Computing En...
 
IoT Workload Distribution Impact Between Edge and Cloud Computing in a Smart ...
IoT Workload Distribution Impact Between Edge and Cloud Computing in a Smart ...IoT Workload Distribution Impact Between Edge and Cloud Computing in a Smart ...
IoT Workload Distribution Impact Between Edge and Cloud Computing in a Smart ...
 
Stream Processing - ThoughtWorks Architecture Group - 2017
Stream Processing - ThoughtWorks Architecture Group - 2017Stream Processing - ThoughtWorks Architecture Group - 2017
Stream Processing - ThoughtWorks Architecture Group - 2017
 
Stream Processing: Uma visão geral - TDC Porto Alegre / FISL 17
Stream Processing: Uma visão geral - TDC Porto Alegre / FISL 17Stream Processing: Uma visão geral - TDC Porto Alegre / FISL 17
Stream Processing: Uma visão geral - TDC Porto Alegre / FISL 17
 
Apache Kafka - Free Friday
Apache Kafka - Free FridayApache Kafka - Free Friday
Apache Kafka - Free Friday
 
Distributed Near Real-Time Processing of Sensor Network Data Flows for Smart ...
Distributed Near Real-Time Processing of Sensor Network Data Flows for Smart ...Distributed Near Real-Time Processing of Sensor Network Data Flows for Smart ...
Distributed Near Real-Time Processing of Sensor Network Data Flows for Smart ...
 
Análise e Caracterização das Novas Ferramentas para Computação em Nuvem
Análise e Caracterização das Novas Ferramentas para Computação em NuvemAnálise e Caracterização das Novas Ferramentas para Computação em Nuvem
Análise e Caracterização das Novas Ferramentas para Computação em Nuvem
 
Utilização de traços de execução para migração de aplicações para a nuvem
Utilização de traços de execução para migração de aplicações para a nuvemUtilização de traços de execução para migração de aplicações para a nuvem
Utilização de traços de execução para migração de aplicações para a nuvem
 

Recently uploaded

WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brandgvaughan
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024Lonnie McRorey
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxLoriGlavin3
 
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLScyllaDB
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .Alan Dix
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your Queries
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your QueriesExploring ChatGPT Prompt Hacks To Maximally Optimise Your Queries
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your QueriesSanjay Willie
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfMounikaPolabathina
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsSergiu Bodiu
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxLoriGlavin3
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxLoriGlavin3
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsPixlogix Infotech
 
Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Manik S Magar
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfLoriGlavin3
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.Curtis Poe
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxLoriGlavin3
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfAddepto
 

Recently uploaded (20)

WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brand
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
 
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQL
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your Queries
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your QueriesExploring ChatGPT Prompt Hacks To Maximally Optimise Your Queries
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your Queries
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdf
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platforms
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and Cons
 
Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdf
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdf
 

A Survey of the State-of-the-art in Event Processing

  • 1. A Survey of the State-of-the-art in Event Processing Otávio M. de Carvalho, Eduardo Roloff, Philippe O. A. Navaux Federal University of Rio Grande do Sul Parallel and Distributed Processing Group {omcarvalho,eroloff,navaux}@inf.ufrgs.br
  • 2. Motivation - We currently live in a world of 2.8 trillion GB - Data production rate is growing 60% by year - Every 60 seconds, 600 new blog posts are published and 34,000 tweets are sent 2
  • 3. Applications of Information Flow Processing 3
  • 4. The History Behind Event Processing Tools ● ● ● The lastest advance in Event Processing can be characterized by two main sub-domains: CEPs and DSMSs systems that were developed in the early 90s Main problem: Today's systems present intersections between these two sub-domains A new naming convention was proposed: Information Flow Processing (IFP) ● Complex Event Processing Systems Those systems focuses on processing event notifications, with a special attention to their ordering relationships, to capture complex event patterns. ● Data Stream Management Systems Those systems focuses mainly on flowing data and the application of transformations over data 4
  • 5. MapReduce is Still Important ● ● The Google's research on MapReduce model was fundamental to the evolution of event processing systems into distributed event processing systems The adoption of Apache Hadoop and his core tools has eased the process of developing distributed systems 5
  • 6. But MapReduce Model Isn't Fast Enought x 6
  • 8. Characterization of Tools Name Release Year DBMS DSMS CEP Distributed Google Photon 2013 • • Walmart Muppet 2012 • • StreamDrill 2012 SAP HANA 2011 Apache Storm 2011 Apache YARN 2011 Apache Flume 2011 • • Apache Kafka 2011 • • Apache S4 2011 • • Apache Chukwa 2010 • • HStreaming 2010 • • AMPLab Spark 2010 • • • VoltDB 2010 • • • Esper 2006 • StreamBase CEP 2003 • SQLstream 2003 • • • • • • • • • • • • • 8
  • 9. Characterization of Tools Name Release Year DBMS DSMS CEP Distributed Google Photon 2013 • • Walmart Muppet 2012 • • StreamDrill 2012 SAP HANA 2011 Apache Storm 2011 Apache YARN 2011 Apache Flume 2011 • • Apache Kafka 2011 • • Apache S4 2011 • • Apache Chukwa 2010 • • HStreaming 2010 • • AMPLab Spark 2010 • • • VoltDB 2010 • • • Esper 2006 StreamBase CEP 2003 SQLstream 2003 • • • • • • • • Legacy Tools • • • • • • • 9
  • 10. Characterization of Tools Name Release Year DBMS DSMS CEP Distributed Google Photon 2013 • • Walmart Muppet 2012 • • StreamDrill 2012 SAP HANA 2011 Apache Storm 2011 Apache YARN 2011 Apache Flume 2011 Apache Kafka 2011 Apache S4 • • In-memory databases • • • • • • • • • • • • 2011 • • Apache Chukwa 2010 • • HStreaming 2010 • • AMPLab Spark 2010 • • • VoltDB 2010 • • • Esper 2006 • StreamBase CEP 2003 • SQLstream 2003 • • • 10
  • 11. Characterization of Tools Name Release Year DBMS DSMS CEP Distributed Google Photon 2013 • • Walmart Muppet 2012 • • StreamDrill 2012 SAP HANA 2011 Apache Storm 2011 Apache YARN 2011 Apache Flume 2011 Apache Kafka • • Flow processing • • • • • • • • • • 2011 • • Apache S4 2011 • • Apache Chukwa 2010 • • HStreaming 2010 • • AMPLab Spark 2010 • • • VoltDB 2010 • • • Esper 2006 • StreamBase CEP 2003 • SQLstream 2003 • • • 11
  • 12. Characterization of Tools Name Release Year DBMS DSMS CEP Distributed Google Photon 2013 • • Walmart Muppet 2012 • • StreamDrill 2012 SAP HANA 2011 Apache Storm 2011 Apache YARN 2011 Apache Flume 2011 • • Apache Kafka 2011 • Apache S4 2011 General-purpose• systems • Apache Chukwa 2010 • • HStreaming 2010 • • AMPLab Spark 2010 • • • VoltDB 2010 • • • Esper 2006 • StreamBase CEP 2003 • SQLstream 2003 • • • • • • • • • • • • • • 12
  • 13. Characterization of Tools Name Release Year DBMS DSMS CEP Distributed Google Photon 2013 • • Walmart Muppet 2012 • • StreamDrill 2012 SAP HANA 2011 Apache Storm 2011 Apache YARN 2011 Apache Flume 2011 • • Apache Kafka 2011 • • Apache S4 2011 • • Apache Chukwa 2010 • • HStreaming 2010 • • AMPLab Spark 2010 • • • VoltDB 2010 • • • Esper 2006 • StreamBase CEP 2003 • SQLstream 2003 • • • Specific to solve one • problem (top-k problem) • • • • • • • • • 13
  • 14. Hadoop Influency Name Based on Hadoop and his core tools Interact with Hadoop Google Photon Walmart Muppet StreamDrill SAP HANA • Apache Storm Apache YARN • Apache Flume • Apache Kafka • Apache S4 Apache Chukwa • HStreaming • AMPLab Spark • VoltDB • Esper StreamBase CEP • SQLstream • 14
  • 15. Challenges in Distributed Event Processing ● How to program? ● In what applications can it be used? ● Now it scales, but is the throughput the same? 15
  • 16. Conclusions ● ● ● MapReduce paper and later contributions have collaborated to the changes into event processing systems Convergences between the tools still are causing naming problems It is not possible to determine if these implementations will converge into greater sets of tools of general-purpose systems or more specialized systems. 16
  • 17. Future Works ● ● ● In our future works, we will select a subset of these tools to analyze further We will evaluate the performance of these selected tools And finally, we will apply one of them in a real HPC application 17
  • 18. A Survey of the State-of-the-art in Event Processing Thanks! Otávio M. de Carvalho, Eduardo Roloff, Philippe O. A. Navaux Federal University of Rio Grande do Sul Parallel and Distributed Processing Group {omcarvalho,eroloff,navaux}@inf.ufrgs.br