This document provides an overview of upcoming features in WSO2 Complex Event Processor 3.0.0. It discusses the Siddhi CEP engine, event processing queries including filters, windows, joins, patterns and event tables. It also covers high availability, persistence, scaling, integration with BAM, and performance comparisons with Esper. The document concludes with a demo of monitoring stock prices and tweets to detect significant stock price changes when a company is highly discussed on Twitter.
Async Debugging A Practical Guide to survive !Mirco Vanini
The speech talk about the specialised tools inside visual studio to survive from async code bugs with special look about how to write a right async code
Practical QML - Key Navigation, Dynamic Language and Theme ChangeBurkhard Stubert
This is the presentation "Practical QML" that I gave at the Qt Developer Days 2013 in Berlin and San Francisco.
What topics entail the liveliest discussions when you talk to the makers of in-vehicle infotainment systems, agricultural driver terminals or set-top boxes about building QML GUIs? No, it is not about "fluid" multi-touch UIs with graphical "wow" effects, but about mundane things like key navigation, theming, and internationalisation. I'll address these three problems in my talk.
First, I’ll discuss how to navigate QML GUIs solely with a rotary knob. By turning or pushing the knob, a "cursor" is moved from one QML component or one screen to another.
Then, I'll show how to switch between different languages and between different themes at run-time.
Async Debugging A Practical Guide to survive !Mirco Vanini
The speech talk about the specialised tools inside visual studio to survive from async code bugs with special look about how to write a right async code
Practical QML - Key Navigation, Dynamic Language and Theme ChangeBurkhard Stubert
This is the presentation "Practical QML" that I gave at the Qt Developer Days 2013 in Berlin and San Francisco.
What topics entail the liveliest discussions when you talk to the makers of in-vehicle infotainment systems, agricultural driver terminals or set-top boxes about building QML GUIs? No, it is not about "fluid" multi-touch UIs with graphical "wow" effects, but about mundane things like key navigation, theming, and internationalisation. I'll address these three problems in my talk.
First, I’ll discuss how to navigate QML GUIs solely with a rotary knob. By turning or pushing the knob, a "cursor" is moved from one QML component or one screen to another.
Then, I'll show how to switch between different languages and between different themes at run-time.
Siddhi: A Second Look at Complex Event Processing ImplementationsSrinath Perera
Today there are so much data being available from sources like sensors (RFIDs, Near Field Communication), web activities, transactions, social networks, etc. Making sense of this avalanche of data requires efficient and fast processing.
Processing of high volume of events to derive higher-level information is a vital part of taking critical decisions, and
Complex Event Processing (CEP) has become one of the most rapidly emerging fields in data processing. e-Science
use-cases, business applications, financial trading applications, operational analytics applications and business activity monitoring applications are some use-cases that directly use CEP. This paper discusses different design decisions associated
with CEP Engines, and proposes some approaches to improve CEP performance by using more stream processing
style pipelines. Furthermore, the paper will discuss Siddhi, a CEP Engine that implements those suggestions. We
present a performance study that exhibits that the resulting CEP Engine—Siddhi—has significantly improved performance.
Primary contributions of this paper are performing a critical analysis of the CEP Engine design and identifying
suggestions for improvements, implementing those improvements
through Siddhi, and demonstrating the soundness of those suggestions through empirical evidence.
To view recording of this webinar please use below URL
http://wso2.com/library/webinars/2015/11/wso2-product-release-webinar-wso2-complex-event-processor-4.0/
In this webinar, Lasantha and Suho will discuss the following key features and improvements in detail:
Integrating WSO2 CEP with Apache Storm to achieve distributed real-time stream processing
Key features of the latest version of Siddhi
New transports that enhances integration capabilities of WSO2 CEP
Creating query templates using execution manager
Using the analytics dashboard to visualize results in real-time
Patterns and practices for real-world event-driven microservicesRachel Reese
Jet.com is an e-commerce startup competing with Amazon. We're heavy users of F#, and have based our architecture around Azure-based event-driven functional microservices. Over the last several months, we've schooled ourselves on what works and what doesn't for F# and microservices. This session will walk you through the lessons we have learned on our way to developing our platform.
C* Summit 2013: Eventual Consistency != Hopeful Consistency by Christos Kalan...DataStax Academy
This session will address Cassandra's tunable consistency model and cover how developers and companies should adopt a more Optimistic Software Design model.
Event Driven Architecture (EDA), November 2, 2006Tim Bass
Event Driven Architecture (EDA), SOA Seminar Crystal City, Virginia, November 2nd, 2006, Tim Bass, CISSP, Principal Global Architect, Director. Co-Chair, Event Processing Reference Architecture Working Group (EPRAWG)
DataStax: Rigorous Cassandra Data Modeling for the Relational Data ArchitectDataStax Academy
With the explosive adoption of Cassandra for online transaction processing by hundreds of Web-scale companies, there is a growing need for a rigorous and practical data modeling approach that ensures sound and efficient schema design. In this talk, we present a query-driven data modeling methodology for Apache Cassandra and provide a side-by-side comparison of traditional Relational database design and Cassandra data modeling. Based on a selected use case, we demonstrate main techniques for designing conceptual, logical, and physical data models in the context of both Relational and Cassandra databases. Finally, we discuss strategies for successful migration from Relational to Cassandra.
A whirlwind tour of Event Driven Architecture, extensibility, Domain Driven Design, Command and Query Responsibility Segregation (CQRS) and Complex Event Processing
Event Sourcing, Domain Driven Design, and Command Query Responsibility Segregation – we hear all of these technologies used together frequently, but how do they actually work together? How do you manage complex co-ordination in a CQRS system?
In this talk, we will discuss a real world example of DDD with ES and CQRS written in F# - a functional first language on the .NET Framework. We’ll take a deep dive into the F# algebraic type system that constructs the domain model. Also, the explanation of the abstract notion of a DDD aggregate root compared to the CQRS implementation of an aggregate root. Finishing with how sagas and triggers facilitate poly-aggregate communication.
Understanding How CQL3 Maps to Cassandra's Internal Data StructureDataStax
CQL3 is the newly ordained, canonical, and best-practices means of interacting with Cassandra. Indeed, the Apache Cassandra documentation itself declares that the Thrift API as “legacy” and recommends that CQL3 be used instead. But I’ve heard several people express their concern over the added layer of abstraction. There seems to be an uncertainty about what’s really happening inside of Cassandra.
In this presentation we will open up the hood and take a look at exactly how Cassandra is treating CQL3 queries. Our first stop will be the Cassandra data structure itself. We will briefly review the concepts of keyspaces, columnfamilies, rows, and columns. And we will explain where this data structure excels and where it does not. Composite rowkeys and columnnames are heavily used with CQL3, so we'll cover their functionality as well.
We will then turn to CQL3. I will demonstrate the basic CQL syntax and show how it maps to the underlying data structure. We will see that CQL actually serves as a sort of best practices interface to the internal Cassandra data structure. We will take this point further by demonstrating CQL3 collections (set, list, and map) and showing how they are really just a creative use of this same internal data structure.
Attendees will leave with a clear, inside-out understanding of CQL3 and will be able use CQL with a confidence that they are following best-practices.
Siddhi: A Second Look at Complex Event Processing ImplementationsSrinath Perera
Today there are so much data being available from sources like sensors (RFIDs, Near Field Communication), web activities, transactions, social networks, etc. Making sense of this avalanche of data requires efficient and fast processing.
Processing of high volume of events to derive higher-level information is a vital part of taking critical decisions, and
Complex Event Processing (CEP) has become one of the most rapidly emerging fields in data processing. e-Science
use-cases, business applications, financial trading applications, operational analytics applications and business activity monitoring applications are some use-cases that directly use CEP. This paper discusses different design decisions associated
with CEP Engines, and proposes some approaches to improve CEP performance by using more stream processing
style pipelines. Furthermore, the paper will discuss Siddhi, a CEP Engine that implements those suggestions. We
present a performance study that exhibits that the resulting CEP Engine—Siddhi—has significantly improved performance.
Primary contributions of this paper are performing a critical analysis of the CEP Engine design and identifying
suggestions for improvements, implementing those improvements
through Siddhi, and demonstrating the soundness of those suggestions through empirical evidence.
To view recording of this webinar please use below URL
http://wso2.com/library/webinars/2015/11/wso2-product-release-webinar-wso2-complex-event-processor-4.0/
In this webinar, Lasantha and Suho will discuss the following key features and improvements in detail:
Integrating WSO2 CEP with Apache Storm to achieve distributed real-time stream processing
Key features of the latest version of Siddhi
New transports that enhances integration capabilities of WSO2 CEP
Creating query templates using execution manager
Using the analytics dashboard to visualize results in real-time
Patterns and practices for real-world event-driven microservicesRachel Reese
Jet.com is an e-commerce startup competing with Amazon. We're heavy users of F#, and have based our architecture around Azure-based event-driven functional microservices. Over the last several months, we've schooled ourselves on what works and what doesn't for F# and microservices. This session will walk you through the lessons we have learned on our way to developing our platform.
C* Summit 2013: Eventual Consistency != Hopeful Consistency by Christos Kalan...DataStax Academy
This session will address Cassandra's tunable consistency model and cover how developers and companies should adopt a more Optimistic Software Design model.
Event Driven Architecture (EDA), November 2, 2006Tim Bass
Event Driven Architecture (EDA), SOA Seminar Crystal City, Virginia, November 2nd, 2006, Tim Bass, CISSP, Principal Global Architect, Director. Co-Chair, Event Processing Reference Architecture Working Group (EPRAWG)
DataStax: Rigorous Cassandra Data Modeling for the Relational Data ArchitectDataStax Academy
With the explosive adoption of Cassandra for online transaction processing by hundreds of Web-scale companies, there is a growing need for a rigorous and practical data modeling approach that ensures sound and efficient schema design. In this talk, we present a query-driven data modeling methodology for Apache Cassandra and provide a side-by-side comparison of traditional Relational database design and Cassandra data modeling. Based on a selected use case, we demonstrate main techniques for designing conceptual, logical, and physical data models in the context of both Relational and Cassandra databases. Finally, we discuss strategies for successful migration from Relational to Cassandra.
A whirlwind tour of Event Driven Architecture, extensibility, Domain Driven Design, Command and Query Responsibility Segregation (CQRS) and Complex Event Processing
Event Sourcing, Domain Driven Design, and Command Query Responsibility Segregation – we hear all of these technologies used together frequently, but how do they actually work together? How do you manage complex co-ordination in a CQRS system?
In this talk, we will discuss a real world example of DDD with ES and CQRS written in F# - a functional first language on the .NET Framework. We’ll take a deep dive into the F# algebraic type system that constructs the domain model. Also, the explanation of the abstract notion of a DDD aggregate root compared to the CQRS implementation of an aggregate root. Finishing with how sagas and triggers facilitate poly-aggregate communication.
Understanding How CQL3 Maps to Cassandra's Internal Data StructureDataStax
CQL3 is the newly ordained, canonical, and best-practices means of interacting with Cassandra. Indeed, the Apache Cassandra documentation itself declares that the Thrift API as “legacy” and recommends that CQL3 be used instead. But I’ve heard several people express their concern over the added layer of abstraction. There seems to be an uncertainty about what’s really happening inside of Cassandra.
In this presentation we will open up the hood and take a look at exactly how Cassandra is treating CQL3 queries. Our first stop will be the Cassandra data structure itself. We will briefly review the concepts of keyspaces, columnfamilies, rows, and columns. And we will explain where this data structure excels and where it does not. Composite rowkeys and columnnames are heavily used with CQL3, so we'll cover their functionality as well.
We will then turn to CQL3. I will demonstrate the basic CQL syntax and show how it maps to the underlying data structure. We will see that CQL actually serves as a sort of best practices interface to the internal Cassandra data structure. We will take this point further by demonstrating CQL3 collections (set, list, and map) and showing how they are really just a creative use of this same internal data structure.
Attendees will leave with a clear, inside-out understanding of CQL3 and will be able use CQL with a confidence that they are following best-practices.
Data to Insight in a Flash: Introduction to Real-Time Analytics with WSO2 Com...WSO2
In this webinar, Sriskandarajah Suhothayan, technical lead at WSO2, will take a closer look at the following use cases:
Natural language processing capabilities of WSO2 CEP: Introducing basic constructs of the CEP
Analyzing a soccer game in Real time: Explaining how complicated scenarios can be implemented
Geo fencing capabilities of WSO2 CEP: Focusing on the CEP’s virtualization support
WSO2Con USA 2015: WSO2 Analytics Platform - The One Stop Shop for All Your Da...WSO2
Today’s highly connected world is flooding businesses with big and fast-moving data. The ability to trawl this data ocean and identify actionable insights can deliver a competitive advantage to any organization. The WSO2 Analytics Platform enables businesses to do just that by providing batch, real-time, interactive and predictive analysis capabilities all in one place.
Introduction to WSO2 Data Analytics PlatformSrinath Perera
WSO2 have had several analytics products: WSO2 BAM and WSO2 CEP for some time (or Big Data products if you prefer the term). We are added WSO2 Machine Learner, a product to create, evaluate, and deploy predictive models and renamed WSO2 BAM to WSO2 DAS ( Data Analytics Server).
The platform let you publish ( collect data) once and process them through batch ( Spark) , realtime ( CEP), search the data ( Lucene) and build machine learning models.
This post describes how all those fit within to a single story.
For more information, see https://iwringer.wordpress.com/2015/03/18/introducing-wso2-analytics-platform-note-for-architects/
Gaining actionable insights in real time enables organizations to grab opportunities and omit threats. Sensing the world, detecting actionable insights, and acting upon them has now become far easier than ever with the advancements of streaming SQL. Below are the topics discussed in this slide.
- Building stream processing applications using streaming SQL
- Deploying and monitoring streaming applications
- Scaling streaming applications
- Building domain specific business UIs
- Visualizing stream processing outputs via dashboards
Flink Forward San Francisco 2018: David Reniz & Dahyr Vergara - "Real-time m...Flink Forward
“Customer experience is the next big battle ground for telcos,” proclaimed recently Amit Akhelikar, Global Director of Lynx Analytics at TM Forum Live! Asia in Singapore. But, how to fight in this battle? A common approach has been to keep “under control” some well-known network quality indicators, like dropped calls, radio access congestion, availability, and so on; but this has proven not to be enough to keep customers happy, like a siege weapon is not enough to conquer a city. But, what if it were possible to know how customers perceive services, at least most demanded ones, like web browsing or video streaming? That would be like a squad of archers ready to battle. And even having that, how to extract value of it and take actions in no time, giving our skilled archers the right targets? Meet CANVAS (Customer And Network Visualization and AnaltyticS), one of the first LATAM implementations of a Flink-based stream processing use case for a telco, which successfully combines leading and innovative technologies like Apache Hadoop, YARN, Kafka, Nifi, Druid and advanced visualizations with Flink core features like non-trivial stateful stream processing (joins, windows and aggregations on event time) and CEP capabilities for alarm generation, delivering a next-generation tool for SOC (Service Operation Center) teams.
Apache Spark Streaming: Architecture and Fault ToleranceSachin Aggarwal
Agenda:
• Spark Streaming Architecture
• How different is Spark Streaming from other streaming applications
• Fault Tolerance
• Code Walk through & demo
• We will supplement theory concepts with sufficient examples
Speakers :
Paranth Thiruvengadam (Architect (STSM), Analytics Platform at IBM Labs)
Profile : https://in.linkedin.com/in/paranth-thiruvengadam-2567719
Sachin Aggarwal (Developer, Analytics Platform at IBM Labs)
Profile : https://in.linkedin.com/in/nitksachinaggarwal
Github Link: https://github.com/agsachin/spark-meetup
Today’s highly connected world is flooding businesses with big and fast-moving data. The ability to trawl this data ocean and identify actionable insights can deliver a competitive advantage to any organization. The WSO2 Analytics Platform enables businesses to do just that by providing batch, real-time, interactive and predictive analysis capabilities all in one place.
In this tutorial we will
* Plug in the WSO2 Analytics Platform to some common business use cases
* Showcase the numerous capabilities of the platform
* Demonstrate how to collect data, analyze, predict and communicate effectively
* Demonstrate how it can analyze integration, security and IoT scenarios
Stick around till the end and you will walk away with the necessary skills to create a winning data strategy for your organization to stay ahead of its competition.
Sumedha Rubasinghe, Director of API Architecture presented this talk at the API Strategy & Practice Conference in Chicago where he illustrated how organisations can analyse who uses their APIs, while understanding how statistics help in capacity planning, deployment, maintain schedules and trend analyses as well as assist the decision-making process. The session discussed scalable collection of statistics for API ecosystems, key design considerations, real-time and offline analysis as well as WSO2’s approach for dealing with these challenges.
Azure event hubs, Stream Analytics & Power BI (by Sam Vanhoutte)Codit
In this presentation Sam gives an overview of how the various Azure IoT Services are used to ingest data (Event Hubs), process and analyze data (Stream Analytics) and visualize data (PowerBI).
This presentation describes progress on ProcessOne new product, OneTeam Media Server. This is a Flash Media Server used to write interactive, collaborative and real time Flash applications.
Accelerate Enterprise Software Engineering with PlatformlessWSO2
Key takeaways:
Challenges of building platforms and the benefits of platformless.
Key principles of platformless, including API-first, cloud-native middleware, platform engineering, and developer experience.
How Choreo enables the platformless experience.
How key concepts like application architecture, domain-driven design, zero trust, and cell-based architecture are inherently a part of Choreo.
Demo of an end-to-end app built and deployed on Choreo.
Less Is More: Utilizing Ballerina to Architect a Cloud Data PlatformWSO2
At its core, the challenge of managing Human Resources data is an integration challenge: estimates range from 2-3 HR systems in use at a typical SMB, up to a few dozen systems implemented amongst enterprise HR departments, and these systems seldom integrate seamlessly between themselves. Providing a multi-tenant, cloud-native solution to integrate these hundreds of HR-related systems, normalize their disparate data models and then render that consolidated information for stakeholder decision making has been a substantial undertaking, but one significantly eased by leveraging Ballerina. In this session, we’ll cover:
The overall software architecture for VHR’s Cloud Data Platform
Critical decision points leading to adoption of Ballerina for the CDP
Ballerina’s role in multiple evolutionary steps to the current architecture
Roadmap for the CDP architecture and plans for Ballerina
WSO2’s partnership in bringing continual success for the CD
The integration landscape is changing rapidly with the introduction of technologies like GraphQL, gRPC, stream processing, iPaaS, and platformless. However, not all existing applications and industries can keep up with these new technologies. Certain industries, like manufacturing, logistics, and finance, still rely on well-established EDI-based message formats. Some applications use XML or CSV with file-based communications, while others have strict on premises deployment requirements. This talk focuses on how Ballerina's built-in integration capabilities can bridge the gap between "old" and "new" technologies, modernizing enterprise applications without disrupting business operations.
Platformless Horizons for Digital AdaptabilityWSO2
In this keynote, Asanka Abeysinghe, CTO,WSO2 will explore the shift towards platformless technology ecosystems and their importance in driving digital adaptability and innovation. We will discuss strategies for leveraging decentralized architectures and integrating diverse technologies, with a focus on building resilient, flexible, and future-ready IT infrastructures. We will also highlight WSO2's roadmap, emphasizing our commitment to supporting this transformative journey with our evolving product suite.
Quantum computers are rapidly evolving and are promising significant advantages in domains like machine learning or optimization, to name but a few areas. In this keynote we sketch the underpinnings of quantum computing, show some of the inherent advantages, highlight some application areas, and show how quantum applications are built.
WSO2CON 2024 - Designing Event-Driven Enterprises: Stories of Transformation
Complex Event Processor 3.0.0 - An overview of upcoming features
1. WSO2 Complex Event Processor
3.0.0
An overview of upcoming features
By
S. Suhothayan
Associate Technical Lead,
Team Lead CEP.
2. About WSO2
• Providing the only complete open source componentized cloud platform
– Dedicated to removing all the stumbling blocks to enterprise agility
– Enabling you to focus on business logic and business value
• Recognized by leading analyst firms as visionaries and leaders
– Gartner cites WSO2 as visionaries in all 3 categories of application
infrastructure
– Forrester places WSO2 in top 2 for API Management
• Global corporation with offices in USA, UK & Sri Lanka
– 200+ employees and growing
• Business model of selling comprehensive support & maintenance for our products
4. Outline
! Scenarios of Event Processing
! WSO2 CEP Server & SOA integrates
! The Siddhi Runtime CEP Engine.
! High availability, Persistence and Scalability of
WSO2 CEP
! How CEP can be combined with Business
Activity Monitoring (BAM).
! Demo
5. CEP Is & Is NOT!
! Is NOT!
o Simple filters
• Simple Event Processing
• E.g. Is this a gold or platinum customer?
o Joining multiple event streams
• Event Stream Processing
! Is !
o Processing multiple event streams
o Identify meaningful patterns among streams
o Using temporal windows
• E.g. Notify if there is a 10% increase in overall trading
activity AND the average price of commodities has
fallen 2% in the last 4 hours
6. WSO2 CEP Server
! Enterprise grade server for CEP runtimes
! Supports several transports (network access)
! Supports several data formats
! Support for multiple CEP runtimes
! Governance
! Monitoring
! Tools (WSO2 Dev Studio)
8. Siddhi CEP Runtime
! Apache License, a java library, Tuple based event model
! Supports distributed processing
! Supports multiple query models
• Based on a SQL-like language
• Supports
! Partitions
! Filters
! Windows
! Joins
! Ordering
! Output Rate Limiting
! and others
9. CEP Event Adaptors
! Is an adaptor for receiving and publishing events
! Has the configurations to connect to external endpoints
! Its many-to-many with CEP engine
10. CEP Event Adaptors
Support for several transports (network access) and data formats
● SOAP/WS-Eventing
XML messages
● REST
JSON messages
● JMS
Map messages
XML messages
Text messages
JSON messages
● SMTP (Email)
Text messages
JSON messages
XML messages
● Thrift - WSO2 data format High Performant Event Capturing &
Delivery Framework supports Java/C/C++/C# via Thrift language
bindings
WSO2 Event
12. CEP Event Builders
• Event builder converts different input event types to a type compatible with the
execution plan. • Subscribes to an Input Event Adaptor to listen for events and sends a converted
WSO2 Event or Basic Event to the Execution Plan • Receives events in different formats and exposes those input streams as stream
definitions • Has a one to many relationship with execution plans in execution plan.
13. CEP Execution Plan
● Is an isolated logical execution unit
● Each execution plan has a set of
Queries
Input & Output stream mappings.
● Its one-to-one with a CEP Backend Runtime Engine
● It deals with Siddhi processing engine.
14. CEP Event Formatter
Event formatter does the inverse – listens to events coming from event
processor and sends converted events to Event adaptors.
There are 5 types of output mapping types are available
● Map
● Text
● WSO2Event
● XML
● JSON
15. Monitoring (Event Tracer & Event Statistics)
! Provides real-time statistical visual illustrations of
request & response counts per time based on CEP
server, execution plan, transport adaptor, event
builder and formatter.
17. Siddhi Queries
! Filters and Projection
! Windows
o Events are processed within temporal windows.
(e.g. for aggregation and joins)
Time window vs. length window.
! Joins - Join two streams
! Event ordering - Identify event sequences and
patterns
! Event Partitions
! Event Tables
18. Filters
from <stream-name> [<conditions>]*
select <attributes>
insert into <stream-name>
! Filters the events by conditions, use to detect simple
condition
! Conditions
o >, <, = , <=, <=, !=
o contains, instanceof
o and, or, not
! Example
from cseEventStream[price >= 20 and symbol==’IBM’]
select symbol, volume
insert into StockQuote
19. Window
from <stream-name> [<conditions>]#window.<window-name>(<
parameters>)
select <attributes>
Insert into <stream-name>
Types of Windows
● (Time | Length) (Sliding| Batch) windows
● Type of aggregate functions
● sum, avg, max, min
Example
from cseEventStream[price >= 20]#window.lengthBatch(50)
select symbol, avg(price) as avgPrice
group by symbol
having avgPrice>50
insert into StockQuote
20. Join
from <stream>#<window> [unidirectional] join <stream>#<window>
on <condition> within <time>
insert into <stream>
! Use to join two streams based on a condition. There
must be at least one window defined
! Unidirectional – event arriving only to the
unidirectional stream triggers join
! Example
from TickEvent[symbol==’IBM’]#window.length(2000)
join NewsEvent#window.time(5 min)
on TickEvent.symbol==NewsEvent.company
select *
insert into JoinStream *
21. Pattern
from [every] <condition> → [every] <condition> … <condition> within
<time>
select <attributes>
insert into StockQuote
! Use to Check condition A happen before/after
condition B.
! Can do iterative checks via “every” keyword.
! Here with “within <time>”, SIddhi emits only events
that are within that time of each other
! Example
from every (a1 = purchase[price < 10] ) -> a2 = purchase [price >10000 and
a1.cardNo==a2.cardNo] within 1 day
select a1.cardNo as cardNo, a2.price as price, a2.place as place
insert into potentialFraud
y1 a1 x1 k5 a2 n7
22. Sequence
from <event-regular-expression> within <time>
select <attributes>
Insert into <stream>
! Regular Expressions supported
o * - Zero or more matches (reluctant).
o + - One or more matches (reluctant).
o ? - Zero or one match (reluctant).
o or – either event
! Here we have to refer events returned by * , + using square
brackets to access a specific occurrence of that event
from a1 = requestOrder[action == "buy"],
b1 = cseEventStream[price > a1.price and symbol==a1.symbol]+,
b2 = cseEventStream[price <b1.price]
select a1. symbol as symbol, b1[0].price as firstPrice, b2.price as orderPrice
insert into purchaseOrder
y1 a1 b1 b1 b2 n7
23. Event Paritions
define <partition-id> by <partition-type> (,<partition-type>)*
Partition types can be one of two types • Variable Partitions - Partitions are created by
the discrete values that are encountered for a
variable
define partition StockSymbol by StockStream.symbol
• Range partitions - Partitions are created
according to predefined ranges of variables
define partition stockVolume by range volume < 10 as 'SMALL',
range volume > 10 and volume < 100 as 'MEDIUM', range
volume > 100 as 'LARGE'
24. Event Tables
define table <table-name> (<attribute-name> <type> {,
<attribute-name> <type>}*) ( from <table-type>.<datasource-name>:<
database-name>.<table-name>)?
Event tables can be used in the same manner as an event stream,
with the difference being that events sent to an event table
being persisted to a data source. CEP supports event tables for • In Memory • Relational
o MySQL
o H2
define table cseEventTable(symbol string, price int, volume float)
from MYSQL.cepDataSource:cepdb.cepEventTable0
25. Working with Event Tables
from <stream> (select <attribute-name> (,<attribute-name>)*
)? insert into <table-name>
Inserts the selected attributes from the input stream into the
event table.
from cseEventCheckStream[symbol==cseEventTable.symbol in
cseEventTable] insert into outStream;
For update and delete
from <stream> update <table-name> (on <condition>)?
from <stream> delete <table-name> (on <condition>)?
26. Performance Results
! We compared Siddhi with Esper, the widely used
opensource CEP engine
! For evaluation, we did setup different queries using both
systems, push events in to the system, and measure the
time till all of them are processed.
! We used Intel(R) Xeon(R) X3440 @2.53GHz , 4 cores 8M
cache 8GB RAM running Debian 2.6.32-5-amd64 Kernel
27. Performance sending event within same JVM
Simple filter without window
from StockTick[prize >6] return symbol, price
28. Performance sending event within same JVM
State machine query for pattern matching
From f=FraudWarningEvent ->
p=PINChangeEvent(accountNumber=f.accountNumber)
return accountNumber;
29. Performance Sending Events over the network
! Here we publihsed data from two client publisher
nodes to the CEP Sever node and sent the triggered
notifications of CEP to a client subscriber node.
! To test the worsecase sinario, 100% of the data
published to CEP is recived at the subscriber node
after processing (No data is filtered)
! We used Intel® Core™ i7-2630QM CPU @ 2.00GHz, 8
cores, 8GB RAM running Ubnthu 12.04, 3.2.0-32-
generic Kernel, for running CEP and used Intel® Core™
i3-2350M CPU @ 2.30GHz, 4 cores, 4GB RAM running
Ubnthu 12.04, 3.2.0-32-generic Kernel, for the three
client nodes.
30. HA/ Persistence
! Ability to recover
runtime state in the
case of a failure.
! Enables queries to span
lifetimes much greater
than server uptime.
! Takes periodic
snapshots and stores
all state information to
a scalable persistence
store (Apache
Cassandra).
! Supports pluggable
persistent stores.
31. Scaling
! Vertically scaling
o Can be distributed as a pipeline
! Horizontally scaling
o Queries like windows, patterns, and Join have
shared states, hence hard to distribute!
o Use distributed cache (Hazelcast) to achieve this
• shared memory and batch processing
32. Event Recording
! Ability to record all/some of the events for
future processing
! Few options
o Publish them to Cassandra cluster using WSO2 data
bridge API or BAM (can process data in Cassandra
with Hadoop using WSO2 BAM).
o Write them to distributed cache
o Custom thrift based event recorder
33. Integration with WSO2 BAM
Data Receiving Data Analyzing Data
Presentation
Data
Publishing
36. Scenario
! Monitoring stock exchange for game changing
moments
! Two input event streams.
o Event stream of Stock Quotes from a stock
exchange
o Event stream of word count on various company
names from twitter pages
! Check whether the last traded price of the
stock has changed significantly(by 2%) within
last minute, and people are twitting about that
company (> 10) within last minute
37.
38. Input events
! Input events are JMS Maps
o Stock Exchange Stream
Map<String, Object> map1 = new HashMap<String, Object>();
map1.put("symbol", "MSFT");
map1.put("price", 26.36);
publisher.publish("AllStockQuotes", map1);
o Twitter Stream
Map<String, Object> map1 = new HashMap<String, Object>();
map1.put("company", "MSFT");
map1.put("wordCount", 8);
publisher.publish("TwitterFeed", map1);
40. Queries
from allStockQuotes[win.time(60000)]
select symbol,price, avg(price) as averagePrice
group by symbol
having ((price > averagePrice*1.02) or (averagePrice*0.98 > price ))
insert into fastMovingStockQuotes
from twitterFeed[win.time(60000)]
select company as company, sum(wordCount) as words
group by company
having (words > 10)
insert into highFrequentTweets
from fastMovingStockQuotes[win.time(60000)] as fastMovingStockQuotes
join highFrequentTweets[win.time(60000)] as highFrequentTweets
on fastMovingStockQuotes.symbol==highFrequentTweets.company
select fastMovingStockQuotes.symbol as company,
fastMovingStockQuotes.averagePrice as amount,
highFrequentTweets.words as words
insert into predictedStockQuotes
41. Alert
! As a Email
Hi
Within last minute, people being twitting about {company}
{words} times, and the last traded price of {company} has
changed by 2% and now being trading at ${amount}.
From
CEP
! As a SMS
42. Useful links
! WSO2 CEP 3.0.0
http://ec2-54-224-94-128.compute-1.amazonaws.com/chunk-02/
N-25_09_2013/wso2cep-3.0.0.zip
! WSO2 CEP http://wso2.com/products/complex-event-processor/
! CEP Performance Info
http://srinathsview.blogspot.com/2013/08/cep-performance-processing-100k-to.
html
! Distributed Processing Sample With Siddhi CEP
and ActiveMQ JMS Broker.
http://suhothayan.blogspot.com/2012/08/distributed-processing-sample-for-wso2.
html
! Creating Custom Data Publishers to BAM/CEP
http://wso2.org/library/articles/2012/07/creating-custom-agents-publish-events-
bamcep
! WSO2 BAM 2.3.0
http://wso2.com/products/business-activity-monitor/
43. Engage with WSO2
• Helping you get the most out of your deployments
• From project evaluation and inception to development
and going into production, WSO2 is your partner in
ensuring 100% project success