SlideShare a Scribd company logo
Parallel Complex Event Processing

Karol Grzegorczyk
03-06-2013
Big Data classification

[http://en.wikipedia.org/wiki/File:3_states_of_data.jpg]
Event-driven architecture
Complex Event Processing solutions
●

Open Source:
–
–

Drools Fusion

–

Storm

–
●

Esper

WSO2 Complex Event Processor

Proprietary software
–

Oracle Complex Event Processing

–

StreamBase Complex Event Processing

–

Informatica RulePoint

–

TIBCO Complex Event Processing
Esper
●

Two editions:
―

Open source library

―

Enterprise server based on Jetty

●

Core component of Esper is a CEP engine.

●

CEP engine is working like database turned upside-down

●

Expressions are defined in Event Processing Language (EPL)
―

Declarative domain specific language

―

Similar with the SQL query language but differs from SQL in its use of views
rather than tables and events instead of records (rows)

―

Views are reused among EPL statements for efficiency!

select * from OrderEvent.win:length(5)
Streams
●

Complex event can be build based on several data streams.
select * from AlertEvent as a, NewsEvent as n
where a.symbol = n.symbol

●

Esper defines two types of data streams:
―

Filter-based event stream
select * from OrderEvent(itemType='shirt')

―

Pattern-based event stream
select * from pattern [
OrderEvent(itemType='shirt') -> OrderEvent(itemType='trousers')]

●

It is possible to join between filter-based and pattern-based streams!

●

Events can be forwarded to others streams using INSERT INTO keywords.

●

It is also possible to update event (using UPDATE keyword) before it applies
to any selecting statements
Views
●

●

●

Events are derived from streams (both filter- and
pattern-based) by views
Default view encloses all events from the stream
since addition of the statement to the engine.
View types:
–

Data windows (e.g. lenght, time)

–

Named windows

–

Extension Views (sorted window, rankied window,
time-order view)

–

Standard views (unique, grouped, size, lastevent)

–

Statistics view (univariate, regression, correlation)
Esper processing

●

●

●

Update listeners and subscriber objects are associated with EPL
statements
By defualt listeners and subscribers are notified when new event that
match EPL query arrive (insert stream)
In addition listeners and subscribers can be notified when some event
that match EPL query is removed from the stream (due to the limit of
particular window)
[Esper Reference]
Filtering

Esper provides two types of filtering:
●

Stream-level filtering
select * from OrderEvent(type= 'shirt')

●

Post-data-window filtering
select * from OrderEvent where type = 'shirt'
[Esper Reference]
[Esper Reference]
Stream-level filtering vs post-data-window filtering

select * from OrderEvent(type= 'shirt')

vs
select * from OrderEvent where type = 'shirt'

The first form is preferred, but still sometimes post-data-window filtering is
desired:
Select one hundred orders and calculate average price of trousers.
select avg(price) from OrderEvent.win:length(100)
where type = 'trousers'
Data Windows
●

Basic windows:
―

―

Length batch window (win:length_batch)

―

Time window (win:time)

―

●

Length window (win:length)

Time batch window (win:time_batch)

Advanced time windows
―

Externally-timed window (win:ext_timed)

―

Externally-timed batch window (win:ext_timed_batch)

―

Time-Length combination batch window (win:time_length_batch)

―

Time-Accumulating window (win:time_accum)

―

Keep-All window (win:keepall)

―

First Length (win:firstlength)

―

First Time (win:firsttime)
[Esper Reference]
[Esper Reference]
Scaling Esper

●

●

According to the documentation Esper exceeds over 500 000 event/s on
a dual CPU 2GHz Intel based hardware, with engine latency below 3
microseconds average (below 10us with more than 99% predictability) on
a VWAP benchmark with 1000 statements registered in the system - this
tops at 70 Mbit/s at 85% CPU usage.
Parallel processing
–

Within one machine
-

–

Context partitions

With multiple machines
-

Partitioned stream

-

Partition by use case
Context
●

Context partition – basic level for locking

●

By default single context partition

●

Context types:
―
―

Hash Segmented

―

Category Segmented

―

Non-overlapping context

―

●

Keyed Segmented

Overlapping context

Nesting context
Keyed Segmented Context

create context ByCustomerAndAccount
partition by custId and account from BankTxn
context ByCustomerAndAccount
select custId, account, sum(amount) from BankTxn
Implicite grouping in select statement.
Hash Segmented Context

Assigns events to context partitions based on result of a hash function and modulo
operation
create context SegmentedByCustomerHash coalesce by hash_code (custId) from
BankTxn granularity 16 preallocate
context SegmentedByCustomerHash
select custId, account, sum(amount) from BankTxn group by custId, account

No implicite grouping in select statement!
Category Segmented Context

Assigns events to context partitions based on the values of one or more event
properties, using a predicate expression(s) to define context partition membership.
create context CategoryByTemp
group temp < 65 as cold,
group temp between 65 and 85 as normal,
group temp > 85 as large
from SensorEvent
context CategoryByTemp
select context.label, count(*) from SensorEvent
Non-overlapping context

Non-overlapping context is created when start condition is meet and ended when end
condition is meet. There is always either one or zero context partions.
create context NineToFive start (0, 9, *, *, *) end (0, 17, *, *, *)
context NineToFive select * from TrafficEvent(speed >= 100)
Overlapping context

This context initiates a new context partition when an initiating condition occurs, and
terminates one or more context partitions when the terminating condition occurs.
create context CtxTrainEnter initiated by TrainEnterEvent as te
terminated after 5 minutes
context CtxTrainEnter select t1 from pattern [t1=TrainEnterEvent ->
timer:interval(5 min) and not TrainLeaveEvent(trainId =
context.te.trainId)]
Context nesting

In case of nested contextx the context declared first controls the
lifecycle of the context(s) declared thereafter.
create context NineToFiveSegmented
context NineToFive start (0, 9, *, *, *) end (0, 17, *, *, *),
context SegmentedByCustomer partition by custId from BankTxn
context NineToFiveSegmented
select custId, account, sum(amount) from BankTxn group by account
Partitioning without context declaration

Grouped data window std:groupwin()
What is the difference between:
select avg(price) from OrderEvent.std:groupwin(itemType).win:length(10)

And
select avg(price) from OrderEvent.win:length(10) group by itemType

?
Parallel processing on multiple machines

Partitioned stream
● Partition by use case
●
[Esper Enterprise Edition Reference]
Thank you

More Related Content

Viewers also liked

Analyzing the CDS market using CGP
Analyzing the CDS market using CGPAnalyzing the CDS market using CGP
Analyzing the CDS market using CGP
Karol Grzegorczyk
 
Standards Based Approach to User Interface Development
Standards Based Approach to User Interface DevelopmentStandards Based Approach to User Interface Development
Standards Based Approach to User Interface Development
Sameer Chavan
 
WSO2 Product Release Webinar: WSO2 Complex Event Processor 4.0
WSO2 Product Release Webinar: WSO2 Complex Event Processor 4.0WSO2 Product Release Webinar: WSO2 Complex Event Processor 4.0
WSO2 Product Release Webinar: WSO2 Complex Event Processor 4.0
WSO2
 
Complex Event Processor 3.0.0 - An overview of upcoming features
Complex Event Processor 3.0.0 - An overview of upcoming features Complex Event Processor 3.0.0 - An overview of upcoming features
Complex Event Processor 3.0.0 - An overview of upcoming features
WSO2
 
WSO2 Complex Event Processor
WSO2 Complex Event ProcessorWSO2 Complex Event Processor
WSO2 Complex Event Processor
Sriskandarajah Suhothayan
 
Complex Event Processing
Complex Event ProcessingComplex Event Processing
Complex Event Processing
Matthew Versaggi
 
UML, OWL and REA based enterprise business model 20110201a
UML, OWL and REA based enterprise business model 20110201aUML, OWL and REA based enterprise business model 20110201a
UML, OWL and REA based enterprise business model 20110201a
Richard Kuo
 
Applying CEP Drools Fusion - Drools jBPM Bootcamps 2011
Applying CEP Drools Fusion - Drools jBPM Bootcamps 2011Applying CEP Drools Fusion - Drools jBPM Bootcamps 2011
Applying CEP Drools Fusion - Drools jBPM Bootcamps 2011
Geoffrey De Smet
 
Patterns and practices for real-world event-driven microservices
Patterns and practices for real-world event-driven microservicesPatterns and practices for real-world event-driven microservices
Patterns and practices for real-world event-driven microservices
Rachel Reese
 
C* Summit 2013: Eventual Consistency != Hopeful Consistency by Christos Kalan...
C* Summit 2013: Eventual Consistency != Hopeful Consistency by Christos Kalan...C* Summit 2013: Eventual Consistency != Hopeful Consistency by Christos Kalan...
C* Summit 2013: Eventual Consistency != Hopeful Consistency by Christos Kalan...
DataStax Academy
 
09 semantic web & ontologies
09 semantic web & ontologies09 semantic web & ontologies
09 semantic web & ontologies
Marina Santini
 
Event Driven Architecture (EDA), November 2, 2006
Event Driven Architecture (EDA), November 2, 2006Event Driven Architecture (EDA), November 2, 2006
Event Driven Architecture (EDA), November 2, 2006
Tim Bass
 
Aaai 2011 event processing tutorial
Aaai 2011 event processing tutorialAaai 2011 event processing tutorial
Aaai 2011 event processing tutorial
Opher Etzion
 
Common ddd pitfalls
Common ddd pitfallsCommon ddd pitfalls
Common ddd pitfalls
Tom Janssens
 
DataStax: Rigorous Cassandra Data Modeling for the Relational Data Architect
DataStax: Rigorous Cassandra Data Modeling for the Relational Data ArchitectDataStax: Rigorous Cassandra Data Modeling for the Relational Data Architect
DataStax: Rigorous Cassandra Data Modeling for the Relational Data Architect
DataStax Academy
 
Esper - CEP Engine
Esper - CEP EngineEsper - CEP Engine
Esper - CEP Engine
Aparna Chaudhary
 
Event Driven Architecture
Event Driven ArchitectureEvent Driven Architecture
Event Driven Architecture
Lourens Naudé
 
Real World Event Sourcing and CQRS
Real World Event Sourcing and CQRSReal World Event Sourcing and CQRS
Real World Event Sourcing and CQRS
Matthew Hawkins
 
From legacy to DDD
From legacy to DDDFrom legacy to DDD
From legacy to DDD
Andrzej Krzywda
 
Siddhi: A Second Look at Complex Event Processing Implementations
Siddhi: A Second Look at Complex Event Processing ImplementationsSiddhi: A Second Look at Complex Event Processing Implementations
Siddhi: A Second Look at Complex Event Processing Implementations
Srinath Perera
 

Viewers also liked (20)

Analyzing the CDS market using CGP
Analyzing the CDS market using CGPAnalyzing the CDS market using CGP
Analyzing the CDS market using CGP
 
Standards Based Approach to User Interface Development
Standards Based Approach to User Interface DevelopmentStandards Based Approach to User Interface Development
Standards Based Approach to User Interface Development
 
WSO2 Product Release Webinar: WSO2 Complex Event Processor 4.0
WSO2 Product Release Webinar: WSO2 Complex Event Processor 4.0WSO2 Product Release Webinar: WSO2 Complex Event Processor 4.0
WSO2 Product Release Webinar: WSO2 Complex Event Processor 4.0
 
Complex Event Processor 3.0.0 - An overview of upcoming features
Complex Event Processor 3.0.0 - An overview of upcoming features Complex Event Processor 3.0.0 - An overview of upcoming features
Complex Event Processor 3.0.0 - An overview of upcoming features
 
WSO2 Complex Event Processor
WSO2 Complex Event ProcessorWSO2 Complex Event Processor
WSO2 Complex Event Processor
 
Complex Event Processing
Complex Event ProcessingComplex Event Processing
Complex Event Processing
 
UML, OWL and REA based enterprise business model 20110201a
UML, OWL and REA based enterprise business model 20110201aUML, OWL and REA based enterprise business model 20110201a
UML, OWL and REA based enterprise business model 20110201a
 
Applying CEP Drools Fusion - Drools jBPM Bootcamps 2011
Applying CEP Drools Fusion - Drools jBPM Bootcamps 2011Applying CEP Drools Fusion - Drools jBPM Bootcamps 2011
Applying CEP Drools Fusion - Drools jBPM Bootcamps 2011
 
Patterns and practices for real-world event-driven microservices
Patterns and practices for real-world event-driven microservicesPatterns and practices for real-world event-driven microservices
Patterns and practices for real-world event-driven microservices
 
C* Summit 2013: Eventual Consistency != Hopeful Consistency by Christos Kalan...
C* Summit 2013: Eventual Consistency != Hopeful Consistency by Christos Kalan...C* Summit 2013: Eventual Consistency != Hopeful Consistency by Christos Kalan...
C* Summit 2013: Eventual Consistency != Hopeful Consistency by Christos Kalan...
 
09 semantic web & ontologies
09 semantic web & ontologies09 semantic web & ontologies
09 semantic web & ontologies
 
Event Driven Architecture (EDA), November 2, 2006
Event Driven Architecture (EDA), November 2, 2006Event Driven Architecture (EDA), November 2, 2006
Event Driven Architecture (EDA), November 2, 2006
 
Aaai 2011 event processing tutorial
Aaai 2011 event processing tutorialAaai 2011 event processing tutorial
Aaai 2011 event processing tutorial
 
Common ddd pitfalls
Common ddd pitfallsCommon ddd pitfalls
Common ddd pitfalls
 
DataStax: Rigorous Cassandra Data Modeling for the Relational Data Architect
DataStax: Rigorous Cassandra Data Modeling for the Relational Data ArchitectDataStax: Rigorous Cassandra Data Modeling for the Relational Data Architect
DataStax: Rigorous Cassandra Data Modeling for the Relational Data Architect
 
Esper - CEP Engine
Esper - CEP EngineEsper - CEP Engine
Esper - CEP Engine
 
Event Driven Architecture
Event Driven ArchitectureEvent Driven Architecture
Event Driven Architecture
 
Real World Event Sourcing and CQRS
Real World Event Sourcing and CQRSReal World Event Sourcing and CQRS
Real World Event Sourcing and CQRS
 
From legacy to DDD
From legacy to DDDFrom legacy to DDD
From legacy to DDD
 
Siddhi: A Second Look at Complex Event Processing Implementations
Siddhi: A Second Look at Complex Event Processing ImplementationsSiddhi: A Second Look at Complex Event Processing Implementations
Siddhi: A Second Look at Complex Event Processing Implementations
 

Similar to Parallel Complex Event Processing

Introducing the WSO2 Complex Event Processor
Introducing the WSO2 Complex Event ProcessorIntroducing the WSO2 Complex Event Processor
Introducing the WSO2 Complex Event Processor
WSO2
 
Kapacitor - Real Time Data Processing Engine
Kapacitor - Real Time Data Processing EngineKapacitor - Real Time Data Processing Engine
Kapacitor - Real Time Data Processing Engine
Prashant Vats
 
WSO2 Product Release Webinar - Introducing the WSO2 Complex Event Processor
WSO2 Product Release Webinar - Introducing the WSO2 Complex Event Processor WSO2 Product Release Webinar - Introducing the WSO2 Complex Event Processor
WSO2 Product Release Webinar - Introducing the WSO2 Complex Event Processor
WSO2
 
Introduction to WSO2 Data Analytics Platform
Introduction to  WSO2 Data Analytics PlatformIntroduction to  WSO2 Data Analytics Platform
Introduction to WSO2 Data Analytics Platform
Srinath Perera
 
Stream Processing with Ballerina
Stream Processing with BallerinaStream Processing with Ballerina
Stream Processing with Ballerina
Sriskandarajah Suhothayan
 
Measurement .Net Performance with BenchmarkDotNet
Measurement .Net Performance with BenchmarkDotNetMeasurement .Net Performance with BenchmarkDotNet
Measurement .Net Performance with BenchmarkDotNet
Vasyl Senko
 
Monitoring InfluxEnterprise
Monitoring InfluxEnterpriseMonitoring InfluxEnterprise
Monitoring InfluxEnterprise
InfluxData
 
Windows Remote Management - EN
Windows Remote Management - ENWindows Remote Management - EN
Windows Remote Management - EN
Kirill Nikolaev
 
Virtual training Intro to Kapacitor
Virtual training  Intro to Kapacitor Virtual training  Intro to Kapacitor
Virtual training Intro to Kapacitor
InfluxData
 
WSO2 Product Release Webinar - WSO2 Complex Event Processor
WSO2 Product Release Webinar - WSO2 Complex Event ProcessorWSO2 Product Release Webinar - WSO2 Complex Event Processor
WSO2 Product Release Webinar - WSO2 Complex Event Processor
WSO2
 
26065613bjmmnnnnnnnnnnnjjjjjjjjjjjjjjjjjj
26065613bjmmnnnnnnnnnnnjjjjjjjjjjjjjjjjjj26065613bjmmnnnnnnnnnnnjjjjjjjjjjjjjjjjjj
26065613bjmmnnnnnnnnnnnjjjjjjjjjjjjjjjjjj
MAHESHV559910
 
Taming event-driven software via formal verification
Taming event-driven software via formal verificationTaming event-driven software via formal verification
Taming event-driven software via formal verification
AdaCore
 
Advance sql - window functions patterns and tricks
Advance sql - window functions patterns and tricksAdvance sql - window functions patterns and tricks
Advance sql - window functions patterns and tricks
Eyal Trabelsi
 
IBM MQ - Monitoring and Managing Hybrid Messaging Environments
IBM MQ - Monitoring and Managing Hybrid Messaging EnvironmentsIBM MQ - Monitoring and Managing Hybrid Messaging Environments
IBM MQ - Monitoring and Managing Hybrid Messaging Environments
MarkTaylorIBM
 
VB Script Overview
VB Script OverviewVB Script Overview
VB Script Overview
Praveen Gorantla
 
Tdd,Ioc
Tdd,IocTdd,Ioc
File000126
File000126File000126
File000126
Desmond Devendran
 
Workshop: Building a Streaming Data Platform on AWS
Workshop: Building a Streaming Data Platform on AWSWorkshop: Building a Streaming Data Platform on AWS
Workshop: Building a Streaming Data Platform on AWS
Amazon Web Services
 
Distributed Computing on PostgreSQL | PGConf EU 2017 | Marco Slot
Distributed Computing on PostgreSQL | PGConf EU 2017 | Marco SlotDistributed Computing on PostgreSQL | PGConf EU 2017 | Marco Slot
Distributed Computing on PostgreSQL | PGConf EU 2017 | Marco Slot
Citus Data
 
Qtp Training Deepti 3 Of 44256
Qtp Training Deepti 3 Of 44256Qtp Training Deepti 3 Of 44256
Qtp Training Deepti 3 Of 44256
Azhar Satti
 

Similar to Parallel Complex Event Processing (20)

Introducing the WSO2 Complex Event Processor
Introducing the WSO2 Complex Event ProcessorIntroducing the WSO2 Complex Event Processor
Introducing the WSO2 Complex Event Processor
 
Kapacitor - Real Time Data Processing Engine
Kapacitor - Real Time Data Processing EngineKapacitor - Real Time Data Processing Engine
Kapacitor - Real Time Data Processing Engine
 
WSO2 Product Release Webinar - Introducing the WSO2 Complex Event Processor
WSO2 Product Release Webinar - Introducing the WSO2 Complex Event Processor WSO2 Product Release Webinar - Introducing the WSO2 Complex Event Processor
WSO2 Product Release Webinar - Introducing the WSO2 Complex Event Processor
 
Introduction to WSO2 Data Analytics Platform
Introduction to  WSO2 Data Analytics PlatformIntroduction to  WSO2 Data Analytics Platform
Introduction to WSO2 Data Analytics Platform
 
Stream Processing with Ballerina
Stream Processing with BallerinaStream Processing with Ballerina
Stream Processing with Ballerina
 
Measurement .Net Performance with BenchmarkDotNet
Measurement .Net Performance with BenchmarkDotNetMeasurement .Net Performance with BenchmarkDotNet
Measurement .Net Performance with BenchmarkDotNet
 
Monitoring InfluxEnterprise
Monitoring InfluxEnterpriseMonitoring InfluxEnterprise
Monitoring InfluxEnterprise
 
Windows Remote Management - EN
Windows Remote Management - ENWindows Remote Management - EN
Windows Remote Management - EN
 
Virtual training Intro to Kapacitor
Virtual training  Intro to Kapacitor Virtual training  Intro to Kapacitor
Virtual training Intro to Kapacitor
 
WSO2 Product Release Webinar - WSO2 Complex Event Processor
WSO2 Product Release Webinar - WSO2 Complex Event ProcessorWSO2 Product Release Webinar - WSO2 Complex Event Processor
WSO2 Product Release Webinar - WSO2 Complex Event Processor
 
26065613bjmmnnnnnnnnnnnjjjjjjjjjjjjjjjjjj
26065613bjmmnnnnnnnnnnnjjjjjjjjjjjjjjjjjj26065613bjmmnnnnnnnnnnnjjjjjjjjjjjjjjjjjj
26065613bjmmnnnnnnnnnnnjjjjjjjjjjjjjjjjjj
 
Taming event-driven software via formal verification
Taming event-driven software via formal verificationTaming event-driven software via formal verification
Taming event-driven software via formal verification
 
Advance sql - window functions patterns and tricks
Advance sql - window functions patterns and tricksAdvance sql - window functions patterns and tricks
Advance sql - window functions patterns and tricks
 
IBM MQ - Monitoring and Managing Hybrid Messaging Environments
IBM MQ - Monitoring and Managing Hybrid Messaging EnvironmentsIBM MQ - Monitoring and Managing Hybrid Messaging Environments
IBM MQ - Monitoring and Managing Hybrid Messaging Environments
 
VB Script Overview
VB Script OverviewVB Script Overview
VB Script Overview
 
Tdd,Ioc
Tdd,IocTdd,Ioc
Tdd,Ioc
 
File000126
File000126File000126
File000126
 
Workshop: Building a Streaming Data Platform on AWS
Workshop: Building a Streaming Data Platform on AWSWorkshop: Building a Streaming Data Platform on AWS
Workshop: Building a Streaming Data Platform on AWS
 
Distributed Computing on PostgreSQL | PGConf EU 2017 | Marco Slot
Distributed Computing on PostgreSQL | PGConf EU 2017 | Marco SlotDistributed Computing on PostgreSQL | PGConf EU 2017 | Marco Slot
Distributed Computing on PostgreSQL | PGConf EU 2017 | Marco Slot
 
Qtp Training Deepti 3 Of 44256
Qtp Training Deepti 3 Of 44256Qtp Training Deepti 3 Of 44256
Qtp Training Deepti 3 Of 44256
 

Recently uploaded

"What does it really mean for your system to be available, or how to define w...
"What does it really mean for your system to be available, or how to define w..."What does it really mean for your system to be available, or how to define w...
"What does it really mean for your system to be available, or how to define w...
Fwdays
 
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
DanBrown980551
 
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin..."$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
Fwdays
 
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptxThe Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
operationspcvita
 
High performance Serverless Java on AWS- GoTo Amsterdam 2024
High performance Serverless Java on AWS- GoTo Amsterdam 2024High performance Serverless Java on AWS- GoTo Amsterdam 2024
High performance Serverless Java on AWS- GoTo Amsterdam 2024
Vadym Kazulkin
 
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-EfficiencyFreshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
ScyllaDB
 
Christine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptxChristine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptx
christinelarrosa
 
ScyllaDB Tablets: Rethinking Replication
ScyllaDB Tablets: Rethinking ReplicationScyllaDB Tablets: Rethinking Replication
ScyllaDB Tablets: Rethinking Replication
ScyllaDB
 
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
Jason Yip
 
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
Edge AI and Vision Alliance
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Hiroshi SHIBATA
 
What is an RPA CoE? Session 1 – CoE Vision
What is an RPA CoE?  Session 1 – CoE VisionWhat is an RPA CoE?  Session 1 – CoE Vision
What is an RPA CoE? Session 1 – CoE Vision
DianaGray10
 
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham HillinQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
LizaNolte
 
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
Alex Pruden
 
Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
AstuteBusiness
 
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid ResearchHarnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Neo4j
 
Demystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through StorytellingDemystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through Storytelling
Enterprise Knowledge
 
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and StandardsLeveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
Neo4j
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Chart Kalyan
 

Recently uploaded (20)

"What does it really mean for your system to be available, or how to define w...
"What does it really mean for your system to be available, or how to define w..."What does it really mean for your system to be available, or how to define w...
"What does it really mean for your system to be available, or how to define w...
 
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
 
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin..."$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
 
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptxThe Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
 
High performance Serverless Java on AWS- GoTo Amsterdam 2024
High performance Serverless Java on AWS- GoTo Amsterdam 2024High performance Serverless Java on AWS- GoTo Amsterdam 2024
High performance Serverless Java on AWS- GoTo Amsterdam 2024
 
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-EfficiencyFreshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
 
Christine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptxChristine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptx
 
ScyllaDB Tablets: Rethinking Replication
ScyllaDB Tablets: Rethinking ReplicationScyllaDB Tablets: Rethinking Replication
ScyllaDB Tablets: Rethinking Replication
 
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
 
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
 
What is an RPA CoE? Session 1 – CoE Vision
What is an RPA CoE?  Session 1 – CoE VisionWhat is an RPA CoE?  Session 1 – CoE Vision
What is an RPA CoE? Session 1 – CoE Vision
 
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham HillinQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
 
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
 
Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
 
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid ResearchHarnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
 
Demystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through StorytellingDemystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through Storytelling
 
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and StandardsLeveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
 

Parallel Complex Event Processing

  • 1. Parallel Complex Event Processing Karol Grzegorczyk 03-06-2013
  • 4. Complex Event Processing solutions ● Open Source: – – Drools Fusion – Storm – ● Esper WSO2 Complex Event Processor Proprietary software – Oracle Complex Event Processing – StreamBase Complex Event Processing – Informatica RulePoint – TIBCO Complex Event Processing
  • 5. Esper ● Two editions: ― Open source library ― Enterprise server based on Jetty ● Core component of Esper is a CEP engine. ● CEP engine is working like database turned upside-down ● Expressions are defined in Event Processing Language (EPL) ― Declarative domain specific language ― Similar with the SQL query language but differs from SQL in its use of views rather than tables and events instead of records (rows) ― Views are reused among EPL statements for efficiency! select * from OrderEvent.win:length(5)
  • 6. Streams ● Complex event can be build based on several data streams. select * from AlertEvent as a, NewsEvent as n where a.symbol = n.symbol ● Esper defines two types of data streams: ― Filter-based event stream select * from OrderEvent(itemType='shirt') ― Pattern-based event stream select * from pattern [ OrderEvent(itemType='shirt') -> OrderEvent(itemType='trousers')] ● It is possible to join between filter-based and pattern-based streams! ● Events can be forwarded to others streams using INSERT INTO keywords. ● It is also possible to update event (using UPDATE keyword) before it applies to any selecting statements
  • 7. Views ● ● ● Events are derived from streams (both filter- and pattern-based) by views Default view encloses all events from the stream since addition of the statement to the engine. View types: – Data windows (e.g. lenght, time) – Named windows – Extension Views (sorted window, rankied window, time-order view) – Standard views (unique, grouped, size, lastevent) – Statistics view (univariate, regression, correlation)
  • 8. Esper processing ● ● ● Update listeners and subscriber objects are associated with EPL statements By defualt listeners and subscribers are notified when new event that match EPL query arrive (insert stream) In addition listeners and subscribers can be notified when some event that match EPL query is removed from the stream (due to the limit of particular window)
  • 10. Filtering Esper provides two types of filtering: ● Stream-level filtering select * from OrderEvent(type= 'shirt') ● Post-data-window filtering select * from OrderEvent where type = 'shirt'
  • 13. Stream-level filtering vs post-data-window filtering select * from OrderEvent(type= 'shirt') vs select * from OrderEvent where type = 'shirt' The first form is preferred, but still sometimes post-data-window filtering is desired: Select one hundred orders and calculate average price of trousers. select avg(price) from OrderEvent.win:length(100) where type = 'trousers'
  • 14. Data Windows ● Basic windows: ― ― Length batch window (win:length_batch) ― Time window (win:time) ― ● Length window (win:length) Time batch window (win:time_batch) Advanced time windows ― Externally-timed window (win:ext_timed) ― Externally-timed batch window (win:ext_timed_batch) ― Time-Length combination batch window (win:time_length_batch) ― Time-Accumulating window (win:time_accum) ― Keep-All window (win:keepall) ― First Length (win:firstlength) ― First Time (win:firsttime)
  • 17. Scaling Esper ● ● According to the documentation Esper exceeds over 500 000 event/s on a dual CPU 2GHz Intel based hardware, with engine latency below 3 microseconds average (below 10us with more than 99% predictability) on a VWAP benchmark with 1000 statements registered in the system - this tops at 70 Mbit/s at 85% CPU usage. Parallel processing – Within one machine - – Context partitions With multiple machines - Partitioned stream - Partition by use case
  • 18. Context ● Context partition – basic level for locking ● By default single context partition ● Context types: ― ― Hash Segmented ― Category Segmented ― Non-overlapping context ― ● Keyed Segmented Overlapping context Nesting context
  • 19. Keyed Segmented Context create context ByCustomerAndAccount partition by custId and account from BankTxn context ByCustomerAndAccount select custId, account, sum(amount) from BankTxn Implicite grouping in select statement.
  • 20. Hash Segmented Context Assigns events to context partitions based on result of a hash function and modulo operation create context SegmentedByCustomerHash coalesce by hash_code (custId) from BankTxn granularity 16 preallocate context SegmentedByCustomerHash select custId, account, sum(amount) from BankTxn group by custId, account No implicite grouping in select statement!
  • 21. Category Segmented Context Assigns events to context partitions based on the values of one or more event properties, using a predicate expression(s) to define context partition membership. create context CategoryByTemp group temp < 65 as cold, group temp between 65 and 85 as normal, group temp > 85 as large from SensorEvent context CategoryByTemp select context.label, count(*) from SensorEvent
  • 22. Non-overlapping context Non-overlapping context is created when start condition is meet and ended when end condition is meet. There is always either one or zero context partions. create context NineToFive start (0, 9, *, *, *) end (0, 17, *, *, *) context NineToFive select * from TrafficEvent(speed >= 100)
  • 23. Overlapping context This context initiates a new context partition when an initiating condition occurs, and terminates one or more context partitions when the terminating condition occurs. create context CtxTrainEnter initiated by TrainEnterEvent as te terminated after 5 minutes context CtxTrainEnter select t1 from pattern [t1=TrainEnterEvent -> timer:interval(5 min) and not TrainLeaveEvent(trainId = context.te.trainId)]
  • 24. Context nesting In case of nested contextx the context declared first controls the lifecycle of the context(s) declared thereafter. create context NineToFiveSegmented context NineToFive start (0, 9, *, *, *) end (0, 17, *, *, *), context SegmentedByCustomer partition by custId from BankTxn context NineToFiveSegmented select custId, account, sum(amount) from BankTxn group by account
  • 25. Partitioning without context declaration Grouped data window std:groupwin() What is the difference between: select avg(price) from OrderEvent.std:groupwin(itemType).win:length(10) And select avg(price) from OrderEvent.win:length(10) group by itemType ?
  • 26. Parallel processing on multiple machines Partitioned stream ● Partition by use case ●

Editor's Notes

  1. CEP assumes multiple sources Synonym of CEP is &apos;event correlation&apos;.
  2. Whereas a typical database stores data, and runs queries against the data, a CEP data stores queries, and runs data through the queries. Language to specify expression-based event pattern matching
  3. Does the &apos;insert into&apos; is used to insert into other streams of events or only into named windows?
  4. Data windows will be discussed later
  5. Stream-level-filtering allows only simple filters
  6. In case of post-data-window filtering update listener is not notified, but window is used (filled) Post-data-window filtering allows more sophisticated filtering
  7. Stream-level filetering has built in optimization. Sometimes post-data-window is neede
  8. The granularity defines the maximum degree of parallelism