This document describes Siddhi, an open source complex event processing (CEP) engine. It discusses the objectives of developing Siddhi as a high performing CEP engine. It outlines Siddhi's architecture including its input/output adapters, compiler, Siddhi manager, and processors. It provides details on Siddhi's implementation and capabilities such as filtering, pattern matching, joins, windows, and aggregators. It also evaluates Siddhi's performance compared to other CEP engines and discusses its deployments, website, documentation, and future plans.

1. A HIGH PERFORMANCE COMPLEX EVENT
PROCESSING ENGINE
http://siddhi.sourceforge.net/

2. Siddhi and NBQSA
• Bronze award under Tertiary Category at the
National Best Quality Software Awards 2011
• Held on 7th October 2011 in Sri Lanka

3. INTRODUCTION
Project Group Members
• Suhothayan S. (Suho)
• Loku Narangoda I.U. (Isuru)
• Chaturanga H.W.G.S (Subash)
• Gajasinghe K.C.B. (Kasun)
Project Supervisors
• Dr. Srinath Perera
• Ms.Vishaka Nanayakkara

4. WHAT IS CEP ?
Complex Event Processing (CEP) is identifying meaningful
patterns, relationships and data abstractions among unrelated
events and fire an immediate response.
Database Applications Event-driven Applications
Query
Ad-hoc queries or requests Continuous standing queries
Paradigm
Latency Seconds, hours, days Milliseconds or less
Tens of thousands of
Data Rate Hundreds of events/sec
events/sec or more
Event
request output
input stream
response stream

5. SCENARIOS OF EVENT PROCESSING
Latency
Months
Days CEP Target Scenarios
Relational Database
hours Operational Analytics
Applications
Applications, e.g.,
Minutes Data Warehousing Logistics, etc.
Applications
Seconds
Web Analytics Applications
100 ms Monitoring Manufacturing Financial
Applications Applications Trading
< 1ms Applications
0 10 100 1000 10000 100000 ~1million
Aggregate Data Rate (Events/sec.)

6. WHY SIDDHI ?

7. CURRENT CEP SOLUTIONS
• S4
• Streambase
• Esper
• TelegraphCQ
• Aurora

8. THE PROBLEMS WITH THEM ?
• Not enough support for complex queries
• Less efficient
- High latency and High memory consumption
• Proprietary

9. SIDDHI CEP
• Project Objectives
• Siddhi Architecture
• What we did?
• Siddhi Implementation
• Siddhi Performance
• Miscellaneous
– Website
– Documentation
– Project Tracker
• Siddhi Deployments

10. SIDDHI CEP
• Project Objectives
• Siddhi Architecture
• What we did?
• Siddhi Implementation
• Siddhi Performance
• Miscellaneous
– Website
– Documentation
– Project Tracker
• Siddhi Deployments

11. PROJECT OBJECTIVES
• Carry out research to come up with effective algorithms
and approaches to implement A High Performing CEP
engine.
• Come up with an effective architecture for a CEP Engine
which can detect events in a computationally
efficient way.

12. FROM RESEARCH
For State Machine
SASE: High-Performance Complex Event Processing
over Streams
For Pipeline Architecture
Aurora: a new model and architecture for data stream
management
For Query Plan Management
Query Processing, Resource Management,
and Approximation in a Data Stream
Management

13. SIDDHI CEP
• Project Objectives
• Siddhi Architecture
• What we did?
• Siddhi Implementation
• Siddhi Performance
• Miscellaneous
– Website
– Documentation
– Project Tracker
• Siddhi Deployments

14. User
Output Adapters
Input Adapters
Compiler
Siddhi Manager
User Input a Query

15. User
Output Adapters
Input Adapters
Compiler
Siddhi Manager
Compiling the query

16. User
Output Adapters
Input Adapters
Compiler
Siddhi Manager
Query get compiled into an object model

17. User
Output Adapters
Input Adapters
Compiler
Siddhi Manager
Query Object Model is parsed to the Siddhi Manager

18. User
Output Adapters
Input Adapters
Compiler
Siddhi Manager
Event arrives to the Input Adapter

19. User
Output Adapters
Input Adapters
Compiler
Siddhi Manager
Convert the Event to a tuple and place it to the input event queue

20. User
Output Adapters
Input Adapters
Compiler
Siddhi Manager
Processor takes the tuple from the queue

21. User
Output Adapters
Input Adapters
Compiler
Siddhi Manager
Executing the queries…Other Events arrives at the same time

22. User
Output Adapters
Input Adapters
Compiler
Siddhi Manager
Non matching event thrown away

23. User
Output Adapters
Input Adapters
Compiler
Siddhi Manager

24. User
Output Adapters
Input Adapters
Compiler
Siddhi Manager
Matching Event creates the output Event

25. User
Output Adapters
Input Adapters
Compiler
Siddhi Manager
Pushing generated Events to the output queue

26. User
Output Adapters
Input Adapters
Compiler
Siddhi Manager
User get notified through output Adapter

27. User
Output Adapters
Input Adapters
Compiler
Siddhi Manager
Same procedure happens again and again…

28. SIDDHI CORE

29. SIDDHI CEP
• Project Objectives
• Siddhi Architecture
• What we did?
• Siddhi Implementation
• Siddhi Performance
• Miscellaneous
– Website
– Documentation
– Project Tracker
• Siddhi Deployments

30. Our PROGRESS
• Initial research
• System Design
• 1st iteration
• Web site
• 2nd iteration
• All major functionalities are implemented
• Profiling and Performance Testing
• Improved Siddhi API
• Siddhi Benchmark
• 3rd iteration
• Adding new features for user requests
• Profiling and Performance Testing

31. SIDDHI CEP
• Project Objectives
• Siddhi Architecture
• What we did?
• Siddhi Implementation
• Siddhi Performance
• Miscellaneous
– Website
– Documentation
– Project Tracker
• Siddhi Deployments

32. SIDDHI FUNCTIONALITIES
• Filter
• State Machine
• Sequence Query
• Pattern Query
• Join Event Streams
• Time Window
• Length Window
• Batch Window

33. API TO WRITE QUERIES
Sample query for Filter
Query query = qf.createQuery(
"StockQuote",
qf.output("price=CSEStream.price"),
qf.inputStream(cseEventStream),
qf.condition("CSEStream.symbol", EQUAL, "IBM")
);

34. API TO WRITE QUERIES
Pattern Queries
For, A* B –
Query query =
qf.createQuery(
"StockQuote",
qf.output("priceA=$0.last.price", "priceB=$1.price"),
qf.inputStreams(cseEventStream, infoStock),
qf.pattern(
qf.star(
qf.condition("CSEStream.price", GREATERTHAN, "$0.prev.price“)
),
qf.condition("CSEStream.price", LESSTHAN, "$0.last.price")
)
);

35. API TO WRITE QUERIES
Aggregators
• Average (avg)
• count
• max
• min
• sum
Query query = qf.createQuery(
"StockQuote",
qf.output( "symbol=CSEStream.symbol“,
"avgPrice=avg(CSEStream.price)",
"count=count(CSEStream.symbol)"),
qf.inputStream(cseEventStream),
qf.condition("CSEStream.symbol", EQUAL, "IBM")
);

36. Development Process
• Test Driven Development (TDD)
• Scrum

37. PERFORMANCE IMPROVEMENTS
• Profiling
– For improving latency, Memory leaks

38. ADDITIONAL FEATURES
• Improvements in the Selection Operator
- using MVEL
• Support conditions on the aggregation functions
-having, group-by
• Event quantifications in sequential patterns
-kleene closure (*)
• Unique function

39. SIDDHI CEP
• Project Objectives
• Siddhi Architecture
• What we did?
• Siddhi Implementation
• Siddhi Performance
• Miscellaneous
– Website
– Documentation
– Project Tracker
• Siddhi Deployments

40. PERFORMANCE TESTING
Details of the Machine
Hardware : Intel- Core 2 Duo 2.10 GHz
Memory 1.9GB
OS : Ubuntu
Release 9.10
Kernel Linux 2.6.31-14-generic
GNOME - 2.28.1

41. PERFORMANCE COMPARISON WITH ESPER
Performance Comparison for a simple filter without a
window
Events Siddhi Esper
(ms) (ms)
10 4 5
100 5.6 9.8
1000 34.5 55.2
10,000 252.6 491
100,000 961.4 1669.2
1000,000 3730.6 5546.8

42. PERFORMANCE COMPARISON WITH ESPER
Performance Comparison for a timed window query for
average calculation for a given symbol
Events Siddhi Esper
(ms) (ms)
10 5 9
100 10.6 16.2
1000 52 109.2
10,000 419.2 1159.2
100,000 1302.8 2724
1000,000 7883.2 11589.8

43. PERFORMANCE COMPARISON WITH ESPER
Performance Comparison for a State machine query
Events Siddhi Esper
(ms) (ms)
10 5.8 30
100 11 58.4
1000 98.6 333
10,000 550.4 2285.6
100,000 1606 9663.2
1000,000 12563 77641

44. SIDDHI CEP
• Project Objectives
• Siddhi Architecture
• What we did?
• Siddhi Implementation
• Siddhi Performance
• Miscellaneous
– Website
– Documentation
– Project Tracker
• Siddhi Deployments

45. WEB SITE
http://siddhi.sourceforge.net/

46. USER GUIDE
http://siddhi.sourceforge.net/userguide/content/userguide.html

47. DEVELOPER GUIDE
http://siddhi.sourceforge.net/devguide/content/devguide.html

48. PROJECT TRACKING PAGE
http://www.ohloh.net/p/siddhi-cep

49. SIDDHI DISTRIBUTION
How other projects can use Siddhi?
The Siddhi jar is now available in Sonatype maven
repository.

50. SIDDHI CEP
• Project Objectives
• Siddhi Architecture
• What we did?
• Siddhi Implementation
• Siddhi Performance
• Miscellaneous
– Website
– Documentation
– Project Tracker
• Siddhi Deployments

51. OPENMRS NCD MODULE
https://wiki.openmrs.org/display/docs/Notifiable
+Condition+Detector+%28NCD%29+Module

52. FUTURE OF SIDDHI
• Pushing Siddhi Project to Apache

53. LITERATURE REVIEW
• T. J. Owens, "Survey of event processing. Technical report, Air Force Research
Laboratory, Information Directorate ," 2007.
• G. Tóth, R. Rácz, J. Pánczél, T. Gergely, A. Beszédes, L. Farkas, L.J. Fülöp, "Survey
on Complex Event Processing and Predictive Analytics," , 2010.
• Alex Cheung, Guoli Li, Balasubramaneyam Maniymaran, Vinod Muthusamy, Reza
Sherafat Kazemzadeh Hans-Arno Jacobsen, "System, The PADRES
Publish/Subscribe,".
• Air Force Research Laboratory, "Survey of Event Processing ," , 2007.
• J. Nagy, "User-Centric Personalized Extensibility for Data-Driven Web Applications ,
IF/AFOSR Minigrant Proposal," , 2007.
• J. Gehrke, B. Panda, M. Riedewald, V. Sharma, and W. White A. Demers, "Cayuga: A
General Purpose Event Monitoring System," , Asolimar, California, January 2007.
• J. Gehrke, M. Hong, M. Riedewald, and W. White A. Demers, "A General Algebra
and Implementation for Monitoring Event Streams," 2005.
• K. Vikram, "FingerLakes: A Distributed Event Stream Monitoring System,".
• D Abadi, D Carney, U. Cetintemel, and M. et al. Cherniack, "Aurora: a data stream
management system," , San Diego, California, 2003.

54. LITERATURE REVIEW
• H. Balakrishnan, M. Balazinska, D. Carney, and U. et al. Cetintemel, "Retrospective on Aurora," vol. 13, no. 4, 2004.
• D. Abadi, Y. Ahmad, and M. Balazinska: U. Cetintemel et al. al., "The Design of the Borealis Stream Processing Engine,"
, 2005.
• S. Zdonik, M. Stonebraker, M. Cherniack, and U. Centintemel et al., "The Aurora and Medusa Projects," , 2003.
• S. Chandrasekaran, O. Cooper, A. Deshpande, and M. Franklin et al., "TelegraphCQ: Continuous Dataflow Processing for
an Uncertain World," , 2003.
• S. Chandrasekaran, O. Cooper, A. Deshpande, M. Franklin, J. Hellerstein, W. Hong, S. Madden, F. Reiss, M. Shah, S.
Krishnamurthy, "TelegraphCQ: An Architectural Status Report," IEEE Data Engineering Bulletin, vol. 26, no. 1, 2003.
• T. Sellis, "Multiple Query Optimization," , 1988.
• N., Sanghai, S., Roy, P., Sudarshan, S. Dalvi, "Pipelining in Multi-Query Optimization," , 2001.
• A., Sudarshan, S., Viswanathan, S. Gupta, "Query Scheduling in Multi Query Optimization," , 2001.
• P., Seshadri, A., Sudarshan, A., Bhobhe, S. Roy, "Efficient and Extensible Algorithms For Multi Query Optimization," ,
2000.
• S. Zdonik. (2006, March) Stream Processing Overview [presentation] Workshop on Event Processing, Hawthorne, New
York.
• M. Liu, L. Ding, E.A. Rundensteiner, and M. Mani M. Li, "Event Stream Processing with Out-of-Order Data Arrival," ,
2007.
• Y. Diao, D. Gyllstrom, and N. Immerman J. Agrawal, "Efficient pattern matching over event streams," , 2008.
• T. Shafeek, "Aurora: A New Model And Architecture For Data Stream Management, B.Tech Seminar Report,
Government Engineering College, Thrissur," 2010.

55. QUESTIONS?

56. THANK YOU!