- The document summarizes a webinar about WSO2 Complex Event Processor (CEP) 2.0.1. - It introduces CEP and the Siddhi runtime engine, describes the WSO2 CEP architecture including brokers, buckets, and management UI. - Performance tests show WSO2 CEP can process a high throughput of events. Features like high availability, persistence, scaling, and integration with BAM are covered. - A demo scenario is presented to detect significant stock price changes and high Twitter mentions within a minute using CEP queries.

1. WSO2 Product Release Webinar
WSO2 Complex Event Processor
2.0.1
Simplifying High Performant Data Processing
S. Suhothayan (Suho)
Software Engineer,
Data Technologies Team.

2. Outline
ƒ What is Complex Event Processing?
ƒ WSO2 CEP Server & SOA integrates
ƒ The Siddhi Runtime CEP Engine.
ƒ High availability, Persistence and Scalability of
WSO2 CEP
ƒ How CEP can be combined with Business
Activity Monitoring (BAM).
ƒ Demo

3. Complex Event Processing ?
Complex Event processing is about
listening to events and detecting patterns in
near real-time without storing all events.
WSO2 Inc. 4

4. CEP Is & Is NOT!
ƒ Is NOT!
o Simple filters
- Simple Event Processing
- E.g. Is this a gold or platinum customer?
o Joining multiple event streams
- Event Stream Processing
ƒ Is !
o Processing multiple event streams
o Identify meaningful patterns among streams
o Useing temporal windows
- E.g. Notify if there is a 10% increase in overall trading
activity AND the average price of commodities has
fallen 2% in the last 4 hours

5. WSO2 CEP Server
ƒ Enterprise grade server for CEP runtimes
ƒ Supports several transports (network access)
ƒ Supports several data formats
ƒ Support for multiple CEP runtimes
ƒ Governance
ƒ Monitoring
ƒ Tools (WSO2 Dev Studio)

6. WSO2 CEP Architecture

7. CEP Brokers
ƒ Is an adaptor for
receiving and
publishing events
ƒ Has the
configurations to
connect to external
endpoints
ƒ Its many-to-many
with CEP engine

8. CEP Brokers
ƒ Support for several transports (network access)
and data formats
o SOAP/WS-Eventing
- XML messages
o REST
- JSON messages
o JMS
- Map messages
- XML messages
- Text messages
o SMTP (Email)
- Text messages
o Thrift - WSO2 data format
High Performant Event Capturing & Delivery Framework supports
Java/C/C++/C# via Thrift language bindings
- WSO2 Event
ƒ & Brokers are pluggable !

9. CEP Buckets
ƒ Is an isolated logical
execution unit
ƒ Each CEP bucket has
a set of
o Queries
o Input & Output
event mappings.
ƒ Its one-to-one with
a CEP Backend
Runtime Engine

10. Opensource CEP Runtimes for Buckets
ƒ Siddhi
o Apache License, a java library, Tuple based event
model
o Supports distributed processing
o Supports multiple query models
- Based on a SQL-like language
- Filters, Windows, Joins, Ordering and others
ƒ Esper, http://esper.codehaus.org (Deprecated)
o GPLv2 License, a Java library, Events can be XML, Map,
Object
o Supports multiple query models
- Based on a SQL-like language
- Filters, Windows, Joins, Ordering and others
ƒ Drools Fusion (Deprecated)
o Apache License, a java library
o Support for temporal reasoning + windows

11. Management UI
ƒ To define,
manage &
monitor
o buckets
o brokers (Data
adopters)

12. Developer Studio UI
ƒ Eclipse based
tool to define
buckets
ƒ Can manage
the
configurations
throughout
the production
lifecycle
ƒ Note: 2.1.0
Still not
support Text
Output
Mapping

13. Monitoring
ƒ Provides real-time statistical visual illustrations of
request & response counts per time based on CEP
server, bucket, broker and topics.

14. Understanding
Siddhi CEP Runtime
Engine

15. Siddhi Queries
ƒ Filters and Projection
ƒ Windows
o Events are processed within temporal windows.
(e.g. for aggregation and joins)
Time window vs. length window.
ƒ Joins
o Join two streams
ƒ Event ordering
o Identify event sequences and patterns

16. Filters
from <stream-name> [<conditions>]*
insert into <stream-name>
ƒ Filters the events by conditions
ƒ Conditions
o >, <, = , <=, <=, !=
o contains, instanceof
o and, or, not
ƒ Example
from cseEventStream[price >= 20 and symbol==’IBM’]
insert into StockQuote symbol, volume

17. Window
from <stream-name> [<conditions>]#window.<window-name>(<parameters>)
Insert [<output-type>] into <stream-name
ƒ Types of Windows
o (Time | Length) (Sliding| Batch) windows
ƒ Type of aggregate functions
o sum, avg, max, min
ƒ Example
from cseEventStream[price >= 20]#window.lengthBatch(50)
insert into StockQuote
symbol, avg(price) as avgPrice
group by symbol
having avgPrice>50

18. Join
from <stream>#<window> [unidirectional] join <stream>#<window>
on <condition> within <time>
insert into <stream>
ƒ Join two streams based on a condition and window
ƒ Unidirectional – event arriving only to the
unidirectional stream triggers join
ƒ Example
from TickEvent[symbol==’IBM’]#window.length(2000)
join NewsEvent#window.time(5 min)
insert into JoinStream *

19. Pattern
from [every] <condition> Æ [every] <condition> … <condition>
within <time>
insert into StockQuote (<attribute-name>* | * )
ƒ Check condition A happen before/after condition B
ƒ Can do iterative checks via “every” keyword.
ƒ Here with “within <time>”, SIddhi emits only events
that are within that time of each other
ƒ Example
from every (a1 = purchase[price < 10] )
Æ a2 = purchase [price >10000 and a1.cardNo==a2.cardNo]
within 1 day
insert into potentialFraud
a1.cardNo as cardNo, a2.price as price, a2.place as place
y1 a1 x1 k5 a2 n7

20. Sequence
from <event-regular-expression> within <time> insert into <stream>
ƒ Regular Expressions supported
o * - Zero or more matches (reluctant).
o + - One or more matches (reluctant).
o ? - Zero or one match (reluctant).
o or – either event
ƒ Here we have to refer events returned by * , + using
square brackets to access a specific occurrence of
that event
from a1 = requestOrder[action == "buy"],
b1 = cseEventStream[price > a1.price and symbol==a1.symbol]+,
b2 = cseEventStream[price <b1.price]
insert into purchaseOrder
a1. symbol as symbol, b1[0].price as firstPrice, b2.price as orderPrice
y1 a1 b1 b1 b2 n7

21. Performance Results
ƒ We compared Siddhi with Esper, the widely used
opensource CEP engine
ƒ For evaluation, we did setup different queries using both
systems, push events in to the system, and measure the
time till all of them are processed.
ƒ We used Intel(R) Xeon(R) X3440 @2.53GHz , 4 cores 8M
cache 8GB RAM running Debian 2.6.32-5-amd64 Kernel

22. Performance Comparison With ESPER
Simple filter without window
from StockTick[prize >6] return symbol, price

23. Performance Comparison With ESPER
State machine query for pattern matching
From f=FraudWarningEvent ->
p=PINChangeEvent(accountNumber=f.accountNumber)
return accountNumber;

24. Performance of WSO2 CEP
ƒ Here we publihsed data from two client publisher
nodes to the CEP Sever node and sent the triggered
notifications of CEP to a client subscriber node.
ƒ To test the worsecase sinario, 100% of the data
published to CEP is recived at the subscriber node
after processing (No data is filtered)
ƒ We used Intel® Core™ i7-2630QM CPU @ 2.00GHz, 8
cores, 8GB RAM running Ubnthu 12.04, 3.2.0-32-
generic Kernel, for running CEP and used Intel® Core™
i3-2350M CPU @ 2.30GHz, 4 cores, 4GB RAM running
Ubnthu 12.04, 3.2.0-32-generic Kernel, for the three
client nodes.

25. Performance of WSO2 CEP
Simple filter without window
from StockTick[prize >6] return symbol, price
WSO2 CEP Throughput
1 2 3 4 5 6 7 8 9 10 50 100
300
250
200
150
100
50
0
Avg 67 135 181 210 212 232 245 250 234 186 187 112
kilo Events/Sec
# Clients

26. HA/ Persistence
ƒ Ability to recover
runtime state in the
case of a failure.
ƒ Enables queries to span
lifetimes much greater
than server uptime.
ƒ Takes periodic
snapshots and stores
all state information to
a scalable persistence
store (Apache
Cassandra).
ƒ Supports pluggable
persistent stores.

27. Scaling
ƒ Vertically scaling
o Can be distributed as a pipeline
ƒ Horizontally scaling
o Queries like windows, patterns, and Join have
shared states, hence hard to distribute!
o Use distributed cache (Hazelcast) to achieve this
- shared memory and batch processing

28. Event Recording
ƒ Ability to record all/some of the events for
future processing
ƒ Few options
o Publish them to Cassandra cluster using WSO2 data
bridge API or BAM (can process data in Cassandra
with Hadoop using WSO2 BAM).
o Write them to distributed cache
o Custom thrift based event recorder

29. WSO2 BAM
Data Receiving Data Analyzing Data
Presentation
Data
Publishing

30. CEP Role within WSO2 Platform

31. DEMO

32. Scenario
ƒ Monitoring stock exchange for game changing
moments
ƒ Two input event streams.
o Event stream of Stock Quotes from a stock
exchange
o Event stream of word count on various company
names from twitter pages
ƒ Check whether the last traded price of the
stock has changed significantly(by 2%) within
last minute, and people are twitting about that
company (> 10) within last minute

34. Example Scenario

35. Input events
ƒ Input events are JMS Maps
o Stock Exchange Stream
Map<String, Object> map1 = new HashMap<String, Object>();
map1.put("symbol", "MSFT");
map1.put("price", 26.36);
publisher.publish("AllStockQuotes", map1);
o Twitter Stream
Map<String, Object> map1 = new HashMap<String, Object>();
map1.put("company", "MSFT");
map1.put("wordCount", 8);
publisher.publish("TwitterFeed", map1);

36. Queries

37. Queries
from allStockQuotes[win.time(60000)]
insert into fastMovingStockQuotes
symbol,price, avg(price) as averagePrice
group by symbol
having ((price > averagePrice*1.02) or (averagePrice*0.98 > price ))
from twitterFeed[win.time(60000)]
insert into highFrequentTweets
company as company, sum(wordCount) as words
group by company
having (words > 10)
from fastMovingStockQuotes[win.time(60000)] as fastMovingStockQuotes
join highFrequentTweets[win.time(60000)] as highFrequentTweets
on fastMovingStockQuotes.symbol==highFrequentTweets.company
insert into predictedStockQuotes
fastMovingStockQuotes.symbol as company,
fastMovingStockQuotes.averagePrice as amount,
highFrequentTweets.words as words

38. Alert
ƒ As a Email
Hi
Within last minute, people being twitting about {company}
{words} times, and the last traded price of {company} has
changed by 2% and now being trading at ${amount}.
From
CEP

39. Useful links
ƒ WSO2 CEP 2.0.1
http://wso2.com/products/complex-event-processor/
ƒ Distributed Processing Sample With Siddhi CEP
and ActiveMQ JMS Broker.
http://suhothayan.blogspot.com/2012/08/distributed-processing-sample-for-wso2.
html
ƒ Creating Custom Data Publishers to BAM/CEP
http://wso2.org/library/articles/2012/07/creating-custom-agents-publish-events-
bamcep
ƒ WSO2 BAM 2.0.1
http://wso2.com/products/business-activity-monitor/

40. Questions?

41. Thank you.