Your SlideShare is downloading. ×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

WSO2 Product Release Webinar - WSO2 Complex Event Processor

578
views

Published on

Published in: Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
578
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
41
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Sriskandarajah Suhothayan Technical Lead Lasantha Fernando 24th Apr. 2014 WSO2 CEP 3.1.0 Product Release Webinar Software Engineer
  • 2. * About the Presenter(s) ๏ Sriskandarajah Suhothayan - suho@wso2.com Suho joined WSO2 in December 2011. He is a technical lead working in the Data technologies team where he leads development on the WSO2 Complex Event Processing Server. In addition to his product development efforts he has provided technology consulting on customer engagements, including customer QuickStart programs mainly focused on SAP integration. ๏ Lasantha Fernando - lasantha@wso2.com Lasantha joined WSO2 in February 2013. He is a software engineer working in the Data technologies team where he mainly focuses on the WSO2 Complex Event Processing Server. In addition to his product development efforts he has provided technology consulting on customer engagements.
  • 3. * About WSO2 ๏ Global enterprise, founded in 2005 by acknowledged leaders in XML, web services technologies, standards and open source ๏ Provides only open source platform-as-a-service for private, public and hybrid cloud deployments ๏ All WSO2 products are 100% open source and released under the Apache License Version 2.0. ๏ Is an Active Member of OASIS, Cloud Security Alliance, OSGi Alliance, AMQP Working Group, OpenID Foundation and W3C. ๏ Driven by Innovation ๏ Launched first open source API Management solution in 2012 ๏ Launched App Factory in 2Q 2013 ๏ Launched Enterprise Store and first open source Mobile solution in 4Q 2013
  • 4. * What WSO2 delivers
  • 5. Outline o Scenarios of Event Processing o What's new in CEP 3.1.0? o WSO2 CEP Server & SOA integration o Improvements in Siddhi Runtime CEP Engine. o High availability of WSO2 CEP o How CEP can be combined with Business Activity Monitoring (BAM). o Demo
  • 6. CEP Is & Is NOT! o Is NOT! o Simple filters ■ Simple Event Processing ■ E.g. Is this a gold or platinum customer? o Joining multiple event streams ■ Event Stream Processing o Is ! o Processing multiple event streams o Identify meaningful patterns among streams o Using temporal windows ■ E.g. Notify if there is a 10% increase in overall trading activity AND the average price of commodities has fallen 2% in the last 4 hours
  • 7. WSO2 CEP Server o Enterprise grade server for CEP runtimes o Supports several transports (network access) o Supports several data formats o Governance o Monitoring o Real-time Dashboard
  • 8. WSO2 CEP Server: Big Picture
  • 9. WSO2 CEP Detailed Architecture
  • 10. CEP Event Adaptors ● Is an adaptor for receiving and publishing events ● Has the configurations to connect to external endpoints ● Its many-to-many with CEP engine
  • 11. CEP Event Adaptors Support for several transports (network access) ● SOAP ● HTTP ● JMS ● SMTP ● SMS ● Thrift ● Kafka Supporting data formats ● XML ● JSON ● Map ● Text ● WSO2Event - WSO2 data format over Thrift for High Performant Event Capturing & Delivery Framework supports Java/C/C++/C# via Thrift language bindings
  • 12. CEP Event Adaptors ● Supports database writes using Map messages ○ Cassandra ○ MYSQL ○ H2 ● Supports custom event adaptors via its pluggable architecture!
  • 13. CEP Event Streams o Event stream is a sequence of events o Events streams have in-flows and out-flows o Inflows can be from o Event builders ■ Converts incoming XML, JSON, etc events to event stream o Execution plans o Outflows are to o Event formatters ■ Converts to event stream to XML, JSON, etc events o Execution plans
  • 14. CEP Execution Plan ● Is an isolated logical execution unit ● Each execution plan imports some of the event streams available in CEP and defines the execution logic using queries and exports the results as output event streams. ● Its one-to-one with a CEP Backend Runtime Engine ● It deals with Siddhi processing engine.
  • 15. Default Event Types o Default events do not need a mapping to be specified in event builder/formatter o Sample default event for XML <events> <event> <metaData> <tenant_id>2</tenant_id> </metaData> <correlationData> <activity_id>ID5</activity_id> </correlationData> <payloadData> <clientPhoneNo>0771117673</clientPhoneNo> <clientName>Mohanadarshan</clientName> <clientResidenceAddress>15, Alexendra road, California</clientResidenceAddress> <clientAccountNo>ACT5673</clientAccountNo> </payloadData> </event> <events> o For Custom events, event mapping should be configured in Event builder and Formatter
  • 16. Monitoring & Debugging (Event Statistics) Provides real-time statistical visual illustrations of request & response counts per time based on CEP server, execution plan, event adaptor, event builder and formatter.
  • 17. Monitoring & Debugging (Event Tracer) Provides the actual message in a textual format before and after processing at each stage of the event flow
  • 18. Real time Dashboard o Powered by WSO2 User Engagement Server (UES) o Gadget generation tool to configure gadgets o Currently supports RDBMS only
  • 19. Writing CEP Queries (using Siddhi Runtime)
  • 20. Siddhi Engine Apache License, a java library, Tuple based event model Supports distributed processing o Filters and Projection o Windows o Events are processed within temporal windows. (e.g. for aggregation and joins) ■ Time window vs. length window. o Joins - Join two streams o Event ordering - Identify event sequences and patterns o Event Partitions o Output Rate Limiting o Event Tables
  • 21. Filters from <stream-name> [<conditions>]* select <attributes> insert into <stream-name> o Filters the events by conditions, use to detect simple condition o Conditions o >, <, = , <=, <=, != o contains, instanceof o and, or, not o Example from cseEventStream[price >= 20 and symbol==’IBM’] select symbol, volume insert into StockQuote
  • 22. Window from <stream-name> [<conditions>]#window.<window-name> (<parameters>) select <attributes> insert into <stream-name> o Types of Windows o (Time | Length) (Sliding| Batch) windows o Types of aggregate functions o sum, avg, max, min o Example from cseEventStream[price >= 20]#window.lengthBatch(50) select symbol, avg(price) as avgPrice group by symbol having avgPrice>50 insert into StockQuote
  • 23. Join from <stream>#<window> [unidirectional] join <stream>#<window> on <condition> within <time> insert into <stream> o Used to join two streams based on a condition. There must be at least one window defined o Unidirectional – event arriving only to the unidirectional stream triggers join o Example from TickEvent[symbol==’IBM’]#window.length(2000) join NewsEvent#window.time(5 min) on TickEvent.symbol==NewsEvent.company select * insert into JoinStream *
  • 24. Pattern from [every] <condition> → [every] <condition> … <condition> within <time> select <attributes> insert into StockQuote o Use to Check condition A happen before/after condition B. o Can do iterative checks via “every” keyword. o Here with “within <time>”, SIddhi emits only events that are within that time of each other o Example from every (a1 = purchase[price < 10] ) -> a2 = purchase [price >10000 and a1.cardNo==a2.cardNo] within 1 day select a1.cardNo as cardNo, a2.price as price, a2.place as place insert into potentialFraud y1 a1 x1 k5 a2 n7
  • 25. Sequence from <event-regular-expression> within <time> select <attributes> insert into <stream> o Regular Expressions supported o * - Zero or more matches (reluctant). o + - One or more matches (reluctant). o ? - Zero or one match (reluctant). o or - either event o Here we have to refer events returned by * , + using square brackets to access a specific occurrence of that event from a1 = requestOrder[action == "buy"], b1 = cseEventStream[price > a1.price and symbol==a1.symbol]+, b2 = cseEventStream[price <b1.price] select a1. symbol as symbol, b1[0].price as firstPrice, b2.price as orderPrice insert into purchaseOrder y1 a1 b1 b1 b2 n7
  • 26. Event Partitions define <partition-id> by <partition-type> (,<partition-type>)* Partition types can be one of two types o Variable Partitions - Partitions are created by the discrete values that are encountered for a variable define partition StockSymbol by StockStream.symbol o Range partitions - Partitions are created according to predefined ranges of variables define partition stockVolume by range volume < 10 as 'SMALL', range volume > 10 && volume < 100 as 'MEDIUM', range volume > 100 as 'LARGE'
  • 27. Event Tables define table <table-name> (<attribute-name> <type> {, <attribute-name> <type>}*) ( from (<parameter-name>=’<value>’)+)? Event tables can be used in the same manner as an event stream, with the difference being that events sent to an event table being persisted to a data source. CEP supports event tables for o In Memory o Relational o MySQL o H2 define table cseEventTable(symbol string, price int, volume float) from ('datasource.name'='cepDataSource', 'table. name'='cepEventTable', 'caching.algorithm'='LRU')
  • 28. Caching Support for Event Tables o Event tables now support several caching algorithms o Basic: A size-based algorithm. Events are cached in a FIFO manner. The oldest event is dropped when the cache is full. o LRU (Least Recently Used): The least recently used event is dropped when the cache is full. o LFU (Least Frequently Used): The least frequently used event is dropped when the cache is full.
  • 29. Working with Event Tables from <stream> (select <attribute-name> (,<attribute-name>)* )? insert into <table-name> Inserts the selected attributes from the input stream into the event table. from cseEventCheckStream[symbol==cseEventTable.symbol in cseEventTable] insert into outStream; For update and delete from <stream> update <table-name> (on <condition>)? from <stream> delete <table-name> (on <condition>)?
  • 30. High Availability o Execution plan specifies the distributed processing mode <executionPlan name="RedundantNodeExecutionPlan" statistics="enable" trace="enable" xmlns="http://wso2.org/carbon/eventprocessor"> ... <siddhiConfiguration> <property name="siddhi.enable.distributed.processing">RedundantNode</property> <property name="siddhi.persistence.snapshot.time.interval.minutes">0</property> </siddhiConfiguration> ... </executionPlan>
  • 31. HA/ Persistence o Option 1: Side by side o Takes 2X hardware, but recommended and and have zero down time o Option 2: Snapshot and restore o Uses less HW o Will lose events between snapshots o Downtime while recovery o ** Some scenarios you can use event tables to keep intermediate state
  • 32. Scaling o Vertically scaling o Can be distributed as a pipeline o Horizontally scaling o Queries like windows, patterns, and Join have shared states, hence hard to distribute!
  • 33. Scaling (Contd.) o Currently users have to setup the pipeline manually (WSO2 team can help) o Work is underway to support above pipeline and distributer operators out of the box.
  • 34. Event Recording o Ability to record all/some of the events for future processing o Few options o Publish them to Cassandra cluster using WSO2 data bridge API or BAM (can process data in Cassandra with Hadoop using WSO2 BAM). o Write them to distributed cache o Custom thrift based event recorder
  • 35. Performance Results o Same JVM Performance (Siddhi with Esper, M means a Million) - 4 core machine o Filters 8M Events/Sec vs Esper 2M o Window 2.5M Events/Sec vs. Esper 1M o Patterns 1.4M Events/Sec about 10X faster than Esper o Over the Network Performance (Using thrift based WSO2 event format) - 8 core machine o Filter 0.25M (or 250K) Event/Sec
  • 36. Integration with WSO2 BAM Data Receiving Data Analyzing Data Presentation Data Publishing
  • 37. CEP Role within WSO2 Platform
  • 38. DEMO
  • 39. Scenario o Monitoring stock exchange for game changing moments o Two input event streams. o Event stream of Stock Quotes from a stock exchange o Event stream of word count on various company names from twitter pages o Check whether the last traded price of the stock has changed significantly(by 2%) within last minute, and people are tweeting about that company (> 10) within last minute
  • 40. Input events o Input events are JMS Maps o Stock Exchange Stream ● Map<String, Object> map1 = new HashMap<String, Object>(); ■ map1.put("symbol", "MSFT"); ■ map1.put("price", 26.36); ● publisher.publish("AllStockQuotes", map1); o Twitter Stream ● Map<String, Object> map1 = new HashMap<String, Object>(); ● map1.put("company", "MSFT"); ● map1.put("wordCount", 8); ● publisher.publish("TwitterFeed", map1);
  • 41. Queries
  • 42. Queries from allStockQuotes[win.time(60000)] select symbol,price, avg(price) as averagePrice group by symbol having ((price > averagePrice*1.02) or (averagePrice*0.98 > price )) insert into fastMovingStockQuotes from twitterFeed[win.time(60000)] select company as company, sum(wordCount) as words group by company having (words > 10) insert into highFrequentTweets from fastMovingStockQuotes[win.time(60000)] as fastMovingStockQuotes join highFrequentTweets[win.time(60000)] as highFrequentTweets on fastMovingStockQuotes.symbol==highFrequentTweets.company select fastMovingStockQuotes.symbol as company, fastMovingStockQuotes.averagePrice as amount, highFrequentTweets.words as words insert into predictedStockQuotes
  • 43. Visualize Results o Create a gadget that reads from MySQL database o The query should be something like SELECT company,words FROM predictedStocks
  • 44. Alert o As an Email (Optional) Hi Within last minute, people being tweeting about {company} {words} times, and the last traded price of {company} has changed by 2% and now being trading at ${amount}. From CEP
  • 45. * More Information ! ๏ WSO2 CEP 3.1.0 http://wso2.com/products/complex-event-processor/ ๏ CEP Performance Info http://srinathsview.blogspot. com/2013/08/cep-performance-processing-100k-to.html ๏ Creating Custom Event Adaptors for CEP 3.1.0 http://wso2. com/library/articles/2013/08/writing-custom-event-adaptors-for- cep-3.0.0/ ๏ Creating Custom Data Publishers to BAM/CEP http://wso2. org/library/articles/2012/07/creating-custom-agents-publish-events- bamcep ๏ Siddhi Query Partitioning http://lasanthatechlog.blogspot. com/2013/08/siddhi-query-partitioning-overview.html ๏ WSO2 BAM 2.4.0 http://wso2.com/products/business-activity-monitor/
  • 46. * Business Model
  • 47. Questions?
  • 48. Contact us !