A HIGH PERFORMANCE COMPLEX EVENT                                PROCESSING ENGINEhttp://siddhi.sourceforge.net/
Siddhi and NBQSA• Bronze award under Tertiary Category at the  National Best Quality Software Awards 2011• Held on 7th Oct...
INTRODUCTIONProject Group Members•   Suhothayan S. (Suho)•   Loku Narangoda I.U. (Isuru)•   Chaturanga H.W.G.S (Subash)•  ...
WHAT IS CEP ?Complex Event Processing (CEP) is identifying meaningfulpatterns, relationships and data abstractions among u...
SCENARIOS OF EVENT PROCESSINGLatencyMonthsDays                                                    CEP Target Scenarios    ...
WHY SIDDHI ?
CURRENT CEP SOLUTIONS•   S4•   Streambase•   Esper•   TelegraphCQ•   Aurora
THE PROBLEMS WITH THEM ?• Not enough support for complex queries• Less efficient     - High latency and High memory consum...
SIDDHI CEP•   Project Objectives•   Siddhi Architecture•   What we did?•   Siddhi Implementation•   Siddhi Performance•   ...
SIDDHI CEP•   Project Objectives•   Siddhi Architecture•   What we did?•   Siddhi Implementation•   Siddhi Performance•   ...
PROJECT OBJECTIVES• Carry out research to come up with effective algorithms  and approaches to implement A High Performing...
FROM RESEARCHFor State Machine  SASE: High-Performance Complex Event Processing  over StreamsFor Pipeline Architecture  Au...
SIDDHI CEP•   Project Objectives•   Siddhi Architecture•   What we did?•   Siddhi Implementation•   Siddhi Performance•   ...
User                                               Output AdaptersInput Adapters                                    Compil...
User                                                   Output AdaptersInput Adapters                                      ...
User                                                           Output AdaptersInput Adapters                              ...
User                                                            Output AdaptersInput Adapters                             ...
User                                                            Output AdaptersInput Adapters                             ...
User                                                             Output AdaptersInput Adapters                            ...
User                                                          Output AdaptersInput Adapters                               ...
User                                                          Output AdaptersInput Adapters                               ...
User                                                        Output AdaptersInput Adapters                                 ...
User                                           Output AdaptersInput Adapters                                Compiler      ...
User                                                           Output AdaptersInput Adapters                              ...
User                                                            Output AdaptersInput Adapters                             ...
User                                                          Output AdaptersInput Adapters                               ...
User                                                           Output AdaptersInput Adapters                              ...
SIDDHI CORE
SIDDHI CEP•   Project Objectives•   Siddhi Architecture•   What we did?•   Siddhi Implementation•   Siddhi Performance•   ...
Our PROGRESS•   Initial research•   System Design•   1st iteration•   Web site•   2nd iteration    • All major functionali...
SIDDHI CEP•   Project Objectives•   Siddhi Architecture•   What we did?•   Siddhi Implementation•   Siddhi Performance•   ...
SIDDHI FUNCTIONALITIES• Filter• State Machine    • Sequence Query    • Pattern Query•   Join Event Streams•   Time Window•...
API TO WRITE QUERIESSample query for FilterQuery query = qf.createQuery(            "StockQuote",            qf.output("pr...
API TO WRITE QUERIESPattern QueriesFor, A* B –Query query =qf.createQuery(     "StockQuote",      qf.output("priceA=$0.las...
API TO WRITE QUERIESAggregators•   Average (avg)•   count•   max•   min•   sumQuery query = qf.createQuery(            "St...
Development Process• Test Driven Development (TDD)• Scrum
PERFORMANCE IMPROVEMENTS• Profiling   – For improving latency, Memory leaks
ADDITIONAL FEATURES• Improvements in the Selection Operator   - using MVEL• Support conditions on the aggregation function...
SIDDHI CEP•   Project Objectives•   Siddhi Architecture•   What we did?•   Siddhi Implementation•   Siddhi Performance•   ...
PERFORMANCE TESTINGDetails of the MachineHardware : Intel- Core 2 Duo 2.10 GHz           Memory 1.9GBOS       : Ubuntu    ...
PERFORMANCE COMPARISON WITH ESPERPerformance Comparison for a simple filter without awindow                               ...
PERFORMANCE COMPARISON WITH ESPERPerformance Comparison for a timed window query foraverage calculation for a given symbol...
PERFORMANCE COMPARISON WITH ESPERPerformance Comparison for a State machine query                                      Eve...
SIDDHI CEP•   Project Objectives•   Siddhi Architecture•   What we did?•   Siddhi Implementation•   Siddhi Performance•   ...
WEB SITE    http://siddhi.sourceforge.net/
USER GUIDEhttp://siddhi.sourceforge.net/userguide/content/userguide.html
DEVELOPER GUIDEhttp://siddhi.sourceforge.net/devguide/content/devguide.html
PROJECT TRACKING PAGE      http://www.ohloh.net/p/siddhi-cep
SIDDHI DISTRIBUTIONHow other projects can use Siddhi?  The Siddhi jar is now available in Sonatype maven  repository.
SIDDHI CEP•   Project Objectives•   Siddhi Architecture•   What we did?•   Siddhi Implementation•   Siddhi Performance•   ...
OPENMRS NCD MODULE  https://wiki.openmrs.org/display/docs/Notifiable  +Condition+Detector+%28NCD%29+Module
FUTURE OF SIDDHI• Pushing Siddhi Project to Apache
LITERATURE REVIEW•   T. J. Owens, "Survey of event processing. Technical report, Air Force Research    Laboratory, Informa...
LITERATURE REVIEW•   H. Balakrishnan, M. Balazinska, D. Carney, and U. et al. Cetintemel, "Retrospective on Aurora," vol. ...
QUESTIONS?
THANK YOU!
Upcoming SlideShare
Loading in …5
×

Siddhi CEP Engine

4,767 views

Published on

http://siddhi.sourceforge.net/

Published in: Technology

Siddhi CEP Engine

  1. 1. A HIGH PERFORMANCE COMPLEX EVENT PROCESSING ENGINEhttp://siddhi.sourceforge.net/
  2. 2. Siddhi and NBQSA• Bronze award under Tertiary Category at the National Best Quality Software Awards 2011• Held on 7th October 2011 in Sri Lanka
  3. 3. INTRODUCTIONProject Group Members• Suhothayan S. (Suho)• Loku Narangoda I.U. (Isuru)• Chaturanga H.W.G.S (Subash)• Gajasinghe K.C.B. (Kasun)Project Supervisors• Dr. Srinath Perera• Ms.Vishaka Nanayakkara
  4. 4. WHAT IS CEP ?Complex Event Processing (CEP) is identifying meaningfulpatterns, relationships and data abstractions among unrelatedevents and fire an immediate response. Database Applications Event-driven Applications Query Ad-hoc queries or requests Continuous standing queries Paradigm Latency Seconds, hours, days Milliseconds or less Tens of thousands of Data Rate Hundreds of events/sec events/sec or more Event request output input stream response stream
  5. 5. SCENARIOS OF EVENT PROCESSINGLatencyMonthsDays CEP Target Scenarios Relational Databasehours Operational Analytics Applications Applications, e.g.,Minutes Data Warehousing Logistics, etc. ApplicationsSeconds Web Analytics Applications100 ms Monitoring Manufacturing Financial Applications Applications Trading< 1ms Applications 0 10 100 1000 10000 100000 ~1million Aggregate Data Rate (Events/sec.)
  6. 6. WHY SIDDHI ?
  7. 7. CURRENT CEP SOLUTIONS• S4• Streambase• Esper• TelegraphCQ• Aurora
  8. 8. THE PROBLEMS WITH THEM ?• Not enough support for complex queries• Less efficient - High latency and High memory consumption• Proprietary
  9. 9. SIDDHI CEP• Project Objectives• Siddhi Architecture• What we did?• Siddhi Implementation• Siddhi Performance• Miscellaneous – Website – Documentation – Project Tracker• Siddhi Deployments
  10. 10. SIDDHI CEP• Project Objectives• Siddhi Architecture• What we did?• Siddhi Implementation• Siddhi Performance• Miscellaneous – Website – Documentation – Project Tracker• Siddhi Deployments
  11. 11. PROJECT OBJECTIVES• Carry out research to come up with effective algorithms and approaches to implement A High Performing CEP engine.• Come up with an effective architecture for a CEP Engine which can detect events in a computationally efficient way.
  12. 12. FROM RESEARCHFor State Machine SASE: High-Performance Complex Event Processing over StreamsFor Pipeline Architecture Aurora: a new model and architecture for data stream managementFor Query Plan Management Query Processing, Resource Management, and Approximation in a Data Stream Management
  13. 13. SIDDHI CEP• Project Objectives• Siddhi Architecture• What we did?• Siddhi Implementation• Siddhi Performance• Miscellaneous – Website – Documentation – Project Tracker• Siddhi Deployments
  14. 14. User Output AdaptersInput Adapters Compiler Siddhi Manager User Input a Query
  15. 15. User Output AdaptersInput Adapters Compiler Siddhi Manager Compiling the query
  16. 16. User Output AdaptersInput Adapters Compiler Siddhi Manager Query get compiled into an object model
  17. 17. User Output AdaptersInput Adapters Compiler Siddhi Manager Query Object Model is parsed to the Siddhi Manager
  18. 18. User Output AdaptersInput Adapters Compiler Siddhi Manager Event arrives to the Input Adapter
  19. 19. User Output AdaptersInput Adapters Compiler Siddhi Manager Convert the Event to a tuple and place it to the input event queue
  20. 20. User Output AdaptersInput Adapters Compiler Siddhi Manager Processor takes the tuple from the queue
  21. 21. User Output AdaptersInput Adapters Compiler Siddhi Manager Executing the queries…Other Events arrives at the same time
  22. 22. User Output AdaptersInput Adapters Compiler Siddhi Manager Non matching event thrown away
  23. 23. User Output AdaptersInput Adapters Compiler Siddhi Manager
  24. 24. User Output AdaptersInput Adapters Compiler Siddhi Manager Matching Event creates the output Event
  25. 25. User Output AdaptersInput Adapters Compiler Siddhi Manager Pushing generated Events to the output queue
  26. 26. User Output AdaptersInput Adapters Compiler Siddhi Manager User get notified through output Adapter
  27. 27. User Output AdaptersInput Adapters Compiler Siddhi Manager Same procedure happens again and again…
  28. 28. SIDDHI CORE
  29. 29. SIDDHI CEP• Project Objectives• Siddhi Architecture• What we did?• Siddhi Implementation• Siddhi Performance• Miscellaneous – Website – Documentation – Project Tracker• Siddhi Deployments
  30. 30. Our PROGRESS• Initial research• System Design• 1st iteration• Web site• 2nd iteration • All major functionalities are implemented • Profiling and Performance Testing • Improved Siddhi API• Siddhi Benchmark• 3rd iteration • Adding new features for user requests • Profiling and Performance Testing
  31. 31. SIDDHI CEP• Project Objectives• Siddhi Architecture• What we did?• Siddhi Implementation• Siddhi Performance• Miscellaneous – Website – Documentation – Project Tracker• Siddhi Deployments
  32. 32. SIDDHI FUNCTIONALITIES• Filter• State Machine • Sequence Query • Pattern Query• Join Event Streams• Time Window• Length Window• Batch Window
  33. 33. API TO WRITE QUERIESSample query for FilterQuery query = qf.createQuery( "StockQuote", qf.output("price=CSEStream.price"), qf.inputStream(cseEventStream), qf.condition("CSEStream.symbol", EQUAL, "IBM") );
  34. 34. API TO WRITE QUERIESPattern QueriesFor, A* B –Query query =qf.createQuery( "StockQuote", qf.output("priceA=$0.last.price", "priceB=$1.price"), qf.inputStreams(cseEventStream, infoStock), qf.pattern( qf.star( qf.condition("CSEStream.price", GREATERTHAN, "$0.prev.price“) ), qf.condition("CSEStream.price", LESSTHAN, "$0.last.price") ));
  35. 35. API TO WRITE QUERIESAggregators• Average (avg)• count• max• min• sumQuery query = qf.createQuery( "StockQuote", qf.output( "symbol=CSEStream.symbol“, "avgPrice=avg(CSEStream.price)", "count=count(CSEStream.symbol)"), qf.inputStream(cseEventStream), qf.condition("CSEStream.symbol", EQUAL, "IBM") );
  36. 36. Development Process• Test Driven Development (TDD)• Scrum
  37. 37. PERFORMANCE IMPROVEMENTS• Profiling – For improving latency, Memory leaks
  38. 38. ADDITIONAL FEATURES• Improvements in the Selection Operator - using MVEL• Support conditions on the aggregation functions -having, group-by• Event quantifications in sequential patterns -kleene closure (*)• Unique function
  39. 39. SIDDHI CEP• Project Objectives• Siddhi Architecture• What we did?• Siddhi Implementation• Siddhi Performance• Miscellaneous – Website – Documentation – Project Tracker• Siddhi Deployments
  40. 40. PERFORMANCE TESTINGDetails of the MachineHardware : Intel- Core 2 Duo 2.10 GHz Memory 1.9GBOS : Ubuntu Release 9.10 Kernel Linux 2.6.31-14-generic GNOME - 2.28.1
  41. 41. PERFORMANCE COMPARISON WITH ESPERPerformance Comparison for a simple filter without awindow Events Siddhi Esper (ms) (ms) 10 4 5 100 5.6 9.8 1000 34.5 55.2 10,000 252.6 491 100,000 961.4 1669.2 1000,000 3730.6 5546.8
  42. 42. PERFORMANCE COMPARISON WITH ESPERPerformance Comparison for a timed window query foraverage calculation for a given symbol Events Siddhi Esper (ms) (ms) 10 5 9 100 10.6 16.2 1000 52 109.2 10,000 419.2 1159.2 100,000 1302.8 2724 1000,000 7883.2 11589.8
  43. 43. PERFORMANCE COMPARISON WITH ESPERPerformance Comparison for a State machine query Events Siddhi Esper (ms) (ms) 10 5.8 30 100 11 58.4 1000 98.6 333 10,000 550.4 2285.6 100,000 1606 9663.2 1000,000 12563 77641
  44. 44. SIDDHI CEP• Project Objectives• Siddhi Architecture• What we did?• Siddhi Implementation• Siddhi Performance• Miscellaneous – Website – Documentation – Project Tracker• Siddhi Deployments
  45. 45. WEB SITE http://siddhi.sourceforge.net/
  46. 46. USER GUIDEhttp://siddhi.sourceforge.net/userguide/content/userguide.html
  47. 47. DEVELOPER GUIDEhttp://siddhi.sourceforge.net/devguide/content/devguide.html
  48. 48. PROJECT TRACKING PAGE http://www.ohloh.net/p/siddhi-cep
  49. 49. SIDDHI DISTRIBUTIONHow other projects can use Siddhi? The Siddhi jar is now available in Sonatype maven repository.
  50. 50. SIDDHI CEP• Project Objectives• Siddhi Architecture• What we did?• Siddhi Implementation• Siddhi Performance• Miscellaneous – Website – Documentation – Project Tracker• Siddhi Deployments
  51. 51. OPENMRS NCD MODULE https://wiki.openmrs.org/display/docs/Notifiable +Condition+Detector+%28NCD%29+Module
  52. 52. FUTURE OF SIDDHI• Pushing Siddhi Project to Apache
  53. 53. LITERATURE REVIEW• T. J. Owens, "Survey of event processing. Technical report, Air Force Research Laboratory, Information Directorate ," 2007.• G. Tóth, R. Rácz, J. Pánczél, T. Gergely, A. Beszédes, L. Farkas, L.J. Fülöp, "Survey on Complex Event Processing and Predictive Analytics," , 2010.• Alex Cheung, Guoli Li, Balasubramaneyam Maniymaran, Vinod Muthusamy, Reza Sherafat Kazemzadeh Hans-Arno Jacobsen, "System, The PADRES Publish/Subscribe,".• Air Force Research Laboratory, "Survey of Event Processing ," , 2007.• J. Nagy, "User-Centric Personalized Extensibility for Data-Driven Web Applications , IF/AFOSR Minigrant Proposal," , 2007.• J. Gehrke, B. Panda, M. Riedewald, V. Sharma, and W. White A. Demers, "Cayuga: A General Purpose Event Monitoring System," , Asolimar, California, January 2007.• J. Gehrke, M. Hong, M. Riedewald, and W. White A. Demers, "A General Algebra and Implementation for Monitoring Event Streams," 2005.• K. Vikram, "FingerLakes: A Distributed Event Stream Monitoring System,".• D Abadi, D Carney, U. Cetintemel, and M. et al. Cherniack, "Aurora: a data stream management system," , San Diego, California, 2003.
  54. 54. LITERATURE REVIEW• H. Balakrishnan, M. Balazinska, D. Carney, and U. et al. Cetintemel, "Retrospective on Aurora," vol. 13, no. 4, 2004.• D. Abadi, Y. Ahmad, and M. Balazinska: U. Cetintemel et al. al., "The Design of the Borealis Stream Processing Engine," , 2005.• S. Zdonik, M. Stonebraker, M. Cherniack, and U. Centintemel et al., "The Aurora and Medusa Projects," , 2003.• S. Chandrasekaran, O. Cooper, A. Deshpande, and M. Franklin et al., "TelegraphCQ: Continuous Dataflow Processing for an Uncertain World," , 2003.• S. Chandrasekaran, O. Cooper, A. Deshpande, M. Franklin, J. Hellerstein, W. Hong, S. Madden, F. Reiss, M. Shah, S. Krishnamurthy, "TelegraphCQ: An Architectural Status Report," IEEE Data Engineering Bulletin, vol. 26, no. 1, 2003.• T. Sellis, "Multiple Query Optimization," , 1988.• N., Sanghai, S., Roy, P., Sudarshan, S. Dalvi, "Pipelining in Multi-Query Optimization," , 2001.• A., Sudarshan, S., Viswanathan, S. Gupta, "Query Scheduling in Multi Query Optimization," , 2001.• P., Seshadri, A., Sudarshan, A., Bhobhe, S. Roy, "Efficient and Extensible Algorithms For Multi Query Optimization," , 2000.• S. Zdonik. (2006, March) Stream Processing Overview [presentation] Workshop on Event Processing, Hawthorne, New York.• M. Liu, L. Ding, E.A. Rundensteiner, and M. Mani M. Li, "Event Stream Processing with Out-of-Order Data Arrival," , 2007.• Y. Diao, D. Gyllstrom, and N. Immerman J. Agrawal, "Efficient pattern matching over event streams," , 2008.• T. Shafeek, "Aurora: A New Model And Architecture For Data Stream Management, B.Tech Seminar Report, Government Engineering College, Thrissur," 2010.
  55. 55. QUESTIONS?
  56. 56. THANK YOU!

×