A HIGH PERFORMANCE COMPLEX EVENT PROCESSING ENGINEhttp://siddhi.sourceforge.net/
Siddhi and NBQSA• Bronze award under Tertiary Category at the National Best Quality Software Awards 2011• Held on 7th October 2011 in Sri Lanka
INTRODUCTIONProject Group Members• Suhothayan S. (Suho)• Loku Narangoda I.U. (Isuru)• Chaturanga H.W.G.S (Subash)• Gajasinghe K.C.B. (Kasun)Project Supervisors• Dr. Srinath Perera• Ms.Vishaka Nanayakkara
WHAT IS CEP ?Complex Event Processing (CEP) is identifying meaningfulpatterns, relationships and data abstractions among unrelatedevents and fire an immediate response. Database Applications Event-driven Applications Query Ad-hoc queries or requests Continuous standing queries Paradigm Latency Seconds, hours, days Milliseconds or less Tens of thousands of Data Rate Hundreds of events/sec events/sec or more Event request output input stream response stream
SCENARIOS OF EVENT PROCESSINGLatencyMonthsDays CEP Target Scenarios Relational Databasehours Operational Analytics Applications Applications, e.g.,Minutes Data Warehousing Logistics, etc. ApplicationsSeconds Web Analytics Applications100 ms Monitoring Manufacturing Financial Applications Applications Trading< 1ms Applications 0 10 100 1000 10000 100000 ~1million Aggregate Data Rate (Events/sec.)
PROJECT OBJECTIVES• Carry out research to come up with effective algorithms and approaches to implement A High Performing CEP engine.• Come up with an effective architecture for a CEP Engine which can detect events in a computationally efficient way.
FROM RESEARCHFor State Machine SASE: High-Performance Complex Event Processing over StreamsFor Pipeline Architecture Aurora: a new model and architecture for data stream managementFor Query Plan Management Query Processing, Resource Management, and Approximation in a Data Stream Management
Our PROGRESS• Initial research• System Design• 1st iteration• Web site• 2nd iteration • All major functionalities are implemented • Profiling and Performance Testing • Improved Siddhi API• Siddhi Benchmark• 3rd iteration • Adding new features for user requests • Profiling and Performance Testing
API TO WRITE QUERIESSample query for FilterQuery query = qf.createQuery( "StockQuote", qf.output("price=CSEStream.price"), qf.inputStream(cseEventStream), qf.condition("CSEStream.symbol", EQUAL, "IBM") );
API TO WRITE QUERIESPattern QueriesFor, A* B –Query query =qf.createQuery( "StockQuote", qf.output("priceA=$0.last.price", "priceB=$1.price"), qf.inputStreams(cseEventStream, infoStock), qf.pattern( qf.star( qf.condition("CSEStream.price", GREATERTHAN, "$0.prev.price“) ), qf.condition("CSEStream.price", LESSTHAN, "$0.last.price") ));
API TO WRITE QUERIESAggregators• Average (avg)• count• max• min• sumQuery query = qf.createQuery( "StockQuote", qf.output( "symbol=CSEStream.symbol“, "avgPrice=avg(CSEStream.price)", "count=count(CSEStream.symbol)"), qf.inputStream(cseEventStream), qf.condition("CSEStream.symbol", EQUAL, "IBM") );
Development Process• Test Driven Development (TDD)• Scrum
PERFORMANCE IMPROVEMENTS• Profiling – For improving latency, Memory leaks
ADDITIONAL FEATURES• Improvements in the Selection Operator - using MVEL• Support conditions on the aggregation functions -having, group-by• Event quantifications in sequential patterns -kleene closure (*)• Unique function
FUTURE OF SIDDHI• Pushing Siddhi Project to Apache
LITERATURE REVIEW• T. J. Owens, "Survey of event processing. Technical report, Air Force Research Laboratory, Information Directorate ," 2007.• G. Tóth, R. Rácz, J. Pánczél, T. Gergely, A. Beszédes, L. Farkas, L.J. Fülöp, "Survey on Complex Event Processing and Predictive Analytics," , 2010.• Alex Cheung, Guoli Li, Balasubramaneyam Maniymaran, Vinod Muthusamy, Reza Sherafat Kazemzadeh Hans-Arno Jacobsen, "System, The PADRES Publish/Subscribe,".• Air Force Research Laboratory, "Survey of Event Processing ," , 2007.• J. Nagy, "User-Centric Personalized Extensibility for Data-Driven Web Applications , IF/AFOSR Minigrant Proposal," , 2007.• J. Gehrke, B. Panda, M. Riedewald, V. Sharma, and W. White A. Demers, "Cayuga: A General Purpose Event Monitoring System," , Asolimar, California, January 2007.• J. Gehrke, M. Hong, M. Riedewald, and W. White A. Demers, "A General Algebra and Implementation for Monitoring Event Streams," 2005.• K. Vikram, "FingerLakes: A Distributed Event Stream Monitoring System,".• D Abadi, D Carney, U. Cetintemel, and M. et al. Cherniack, "Aurora: a data stream management system," , San Diego, California, 2003.
LITERATURE REVIEW• H. Balakrishnan, M. Balazinska, D. Carney, and U. et al. Cetintemel, "Retrospective on Aurora," vol. 13, no. 4, 2004.• D. Abadi, Y. Ahmad, and M. Balazinska: U. Cetintemel et al. al., "The Design of the Borealis Stream Processing Engine," , 2005.• S. Zdonik, M. Stonebraker, M. Cherniack, and U. Centintemel et al., "The Aurora and Medusa Projects," , 2003.• S. Chandrasekaran, O. Cooper, A. Deshpande, and M. Franklin et al., "TelegraphCQ: Continuous Dataflow Processing for an Uncertain World," , 2003.• S. Chandrasekaran, O. Cooper, A. Deshpande, M. Franklin, J. Hellerstein, W. Hong, S. Madden, F. Reiss, M. Shah, S. Krishnamurthy, "TelegraphCQ: An Architectural Status Report," IEEE Data Engineering Bulletin, vol. 26, no. 1, 2003.• T. Sellis, "Multiple Query Optimization," , 1988.• N., Sanghai, S., Roy, P., Sudarshan, S. Dalvi, "Pipelining in Multi-Query Optimization," , 2001.• A., Sudarshan, S., Viswanathan, S. Gupta, "Query Scheduling in Multi Query Optimization," , 2001.• P., Seshadri, A., Sudarshan, A., Bhobhe, S. Roy, "Efficient and Extensible Algorithms For Multi Query Optimization," , 2000.• S. Zdonik. (2006, March) Stream Processing Overview [presentation] Workshop on Event Processing, Hawthorne, New York.• M. Liu, L. Ding, E.A. Rundensteiner, and M. Mani M. Li, "Event Stream Processing with Out-of-Order Data Arrival," , 2007.• Y. Diao, D. Gyllstrom, and N. Immerman J. Agrawal, "Efficient pattern matching over event streams," , 2008.• T. Shafeek, "Aurora: A New Model And Architecture For Data Stream Management, B.Tech Seminar Report, Government Engineering College, Thrissur," 2010.