Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

WSO2 Product Release Webinar: WSO2 Complex Event Processor 4.0

795 views

Published on

To view recording of this webinar please use below URL

http://wso2.com/library/webinars/2015/11/wso2-product-release-webinar-wso2-complex-event-processor-4.0/

In this webinar, Lasantha and Suho will discuss the following key features and improvements in detail:

Integrating WSO2 CEP with Apache Storm to achieve distributed real-time stream processing
Key features of the latest version of Siddhi
New transports that enhances integration capabilities of WSO2 CEP
Creating query templates using execution manager
Using the analytics dashboard to visualize results in real-time

Published in: Technology
  • Be the first to comment

WSO2 Product Release Webinar: WSO2 Complex Event Processor 4.0

  1. 1. WSO2 Complex Event Processor 4.0 Sriskandarajah Suhothayan Technical Lead Product Release Webinar Lasantha Fernando Senior Software Engineer
  2. 2. WSO2 Analytics Platform
  3. 3. WSO2 Analytics Platform
  4. 4. Data Processing Pipeline
  5. 5. Realtime Analytics
  6. 6. What is WSO2 CEP ?
  7. 7. New in CEP 4.0 o Re Architecture of the CEP Server o Rewrite of Siddhi 3.0 o Scalable Distributed Processing with Apache Storm o Better High Availability support o Domain Specific Execution Manager o Realtime Dashboard o Improved usability
  8. 8. Event Flow of WSO2 CEP 4.0 CEP 3.x Input Adaptor + Event Builder = CEP 4.0 Event Receiver CEP 3.x Output Adaptor + Event Formatter = CEP 4.0 Event Publisher
  9. 9. Receiving & Notifying Events
  10. 10. *Supports custom event receivers via its pluggable architecture! Highly Pluggable Event Receiver Architecture
  11. 11. Highly Pluggable Event Publisher Architecture *Supports custom event publishers via its pluggable architecture!
  12. 12. Execution of Queries
  13. 13. What’s new in Siddhi 3.0 o New architecture o Optimized for better performance o Improved Query API and Compiler (Antlr 4) o Integration of Disruptor o Better Extendability o Annotation support o Native support for distributed processing via Partition Grouping o Event Trigger Support
  14. 14. What’s new in Siddhi 3.0 ... o Inbuilt Scripting support o JavaScript, R & Scala o Event Table with o RDBMS & Data Analytics Server Tables o New Window support o Cron & Sorting o Realtime Machine learning o R, WSO2ML & PMML o Rich set of Siddhi Extensions
  15. 15. define stream StockStream (symbol string, price double, volume int); partition by (symbol of StockStream) begin from t1=StockStream, t2=StockStream [(t2[last] is null and t1.price < price) or (t2[last].price < price)]+ within 5 min select t1.price as initialPrice, t2[last].price as finalPrice,t1.symbol insert into IncreasingMyStockPriceStream end; Siddhi Query (Partition)
  16. 16. define table CardUserTable (name string, cardNum long) ; @from(eventtable = 'rdbms' , datasource.name = ‘CardDataSource’ , table.name = ‘UserTable’, caching.algorithm’=‘LRU’) define table CardUserTable (name string, cardNum long) Cache types supported o Basic: A size-based algorithm based on FIFO. o LRU (Least Recently Used): The least recently used event is dropped when cache is full. o LFU (Least Frequently Used): The least frequently used event is dropped when cache is full. Siddhi Query (Event Table) Supported for RDBMS, In- Memory, Analytics Table
  17. 17. define function concatJ [JavaScript] return string { var str1 = data[0]; var str2 = data[1]; var str3 = data[2]; var res = str1.concat(str2,str3); return res; }; from cseEventStream select price , concatJ(symbol,' ',price) as concatStr group by volume insert into mailOutput; Siddhi Query (Scripting) Supported for Javascript, Scala and R. Can be extended for other languages
  18. 18. o Function extension o Aggregator extension o Window extension o Stream Processor extension define stream SalesStream (brand string, price double, currency string); from SalesStream select brand, custom:toUSD(price, currency) as priceInUSD insert into OutputStream ; Referred with namespaces Siddhi Query (Extension) ...
  19. 19. Provides support for common functionalities across multiple domains out of the box o geo: Geographical processing o nlp: Natural language Processing (with Stanford NLP) o ml: Running machine learning models of WSO2 Machine Lerner o pmml: Running PMML models learnt by R o timeseries: Regression and time series o math: Mathematical operations o str: String operations o regex: Regular expression o ... New Siddhi Extensions
  20. 20. Siddhi Try-it Tool Allows users to o Try out different queries o Validate queries by sending events
  21. 21. Event Simulator Allows users to test execution environment and simulate complex execution scenarios o Ability to send events to a stream without configuring event receivers o Send events o Directly via UI o Upload a file and send/replay events o Point to a database table and play those events o Configurable delay between events when sending via file/database
  22. 22. Event Simulator
  23. 23. Ideal for business users, Form based UI to create/edit/delete execution logic o Allows to create a predefined execution plan template to a domain o Which can be managed via UI Domain Specific Execution Manager
  24. 24. Event Flow o Visualizes the event flow to provide an overall context
  25. 25. Scaling & HA
  26. 26. High Availability with WSO2 CEP 4.0
  27. 27. Scalability: Why not use existing CEPs ... Spark Streaming o Supports distributed processing o Runs micro batches o Does not support pattern & sequence detection Apache Storm o Supports distributed processing o Stream processing engine
  28. 28. Scalability: Why not use Apache Storm ? Advantages o Supports distributed processing o Supports Partitioning o Extendable o Opensource Disadvantages o Need to write Java code o Need to start from basic principles ( & data structures ) o Adoption for change is slow o No support to govern artifacts
  29. 29. WSO2 CEP 4.0 Scalability Distributed Realtime = Siddhi + Advantages over Apache Storm o No need to write Java code (Supports SQL like query language) o No need to start from basic principles (Supports high level language) o Adoption for change is fast o Govern artifacts using Toolboxes o etc ...
  30. 30. How we scale ?
  31. 31. Scaling with Storm Handling Stateless & Stateful Queries
  32. 32. Siddhi QL define stream StockStream (symbol string, volume int, price double); @name(‘Filter Query’) from StockStream[price > 75] select * insert into HighPriceStockStream ; @name(‘Window Query’) from HighPriceStockStream#window.time(10 min) select symbol, sum(volume) as sumVolume insert into ResultStockStream ;
  33. 33. Siddhi QL - with partition define stream StockStream (symbol string, volume int, price double); @name(‘Filter Query’) from StockStream[price > 75] select * insert into HighPriceStockStream ; @name(‘Window Query’) partition with (symbol of HighPriceStockStream) begin from HighPriceStockStream#window.time(10 min) select symbol, sum(volume) as sumVolume insert into ResultStockStream ; end;
  34. 34. Siddhi QL - distributed define stream StockStream (symbol string, volume int, price double); @name(Filter Query’) @dist(parallel= ‘3') from StockStream[price > 75] select * insert into HightPriceStockStream ; @name(‘Window Query’) @dist(parallel= ‘2') partition with (symbol of HighPriceStockStream) begin from HighPriceStockStream#window.time(10 min) select symbol, sum(volume) as sumVolume insert into ResultStockStream ; end;
  35. 35. Distributed Execution on Storm UI
  36. 36. Realtime Dashboard
  37. 37. Realtime Dashboard Provides an overall idea on what's going on o Dashboard using o Google Gadget o HTML5 + javascripts o Support custom gadgets and dashboards o Gadget generation with o D3 and Vega
  38. 38. Gadget Generation Tool Customize with few clicks … Support realtime and stored data visualization How ? ○ Start with data in tabular format ○ Select chart type (Bar, Line, Map, ...) ○ Map columns to dimensions (X Y, color, point size, … ) ○ Also do drill downs
  39. 39. Geo Dashboard With configurable alerting & Monitoring capabilities from UI. http://wso2.com/library/articles/2015/01/article-geo-spatial-data-analysis-using-wso2- complex-event-processor-0/
  40. 40. Demo
  41. 41. Questions?
  42. 42. More Information o WSO2 CEP 4.0.0 download - http://wso2.com/products/complex-event-processor/ o Product documentation - https://docs.wso2. com/display/CEP400/WSO2+Complex+Event+Processor+Documentation o WSO2 Analytics Platform - http://wso2.com/analytics o Quick Start Guide doc - https://docs.wso2. com/display/CEP400/Quick+Start+Guide o Quick Start Guide video - https://www.youtube.com/playlist? list=PLp0TUr0bmhX5v5yBa0TINUktuF-_eDPC_ o Pre Release Webinar on CEP 4.0 - http://wso2.com/library/webinars/2015/07/real-time-analytics-from-data-to-actions- in-milliseconds/
  43. 43. Contact us !

×