LiSEP: a Lightweight and Extensible tool for
Complex Event Processing
Ivan Zappia, David Parlanti, Federica Paganelli
Nati...
ReferencesReferences
1/19
Ivan Zappia, David Parlanti, Federica Paganelli / LiSEP: a Lightweight and Extensible tool for C...
ContextContext
1/19
Ivan Zappia, David Parlanti, Federica Paganelli / LiSEP: a Lightweight and Extensible tool for Complex...
Complex Event ProcessingComplex Event Processing
A family of technologies concurring at the elaboration of large amount of...
Research goalResearch goal
This paper proposes a novel Complex Event
Processing engine conceived with extensibility,
porta...
Requirements analysisRequirements analysis
Functional requirements:
- system should allow event patterns definition and me...
Event Processing Language (1/2)Event Processing Language (1/2)
High level and SQL-like language; clauses have been “extend...
Event Processing Language (2/2)Event Processing Language (2/2)
To interpret the EPL language and translate statements in o...
Design choicesDesign choices
Non-functional requirements reception:
• Java programming language was adopted to match the
p...
Staged Event-Driven ArchitectureStaged Event-Driven Architecture
8/19
• Idea: decomposition of a complex event-driven syst...
Layered architectureLayered architecture
Hardware
Operating System
Java Virtual Machine
SEDA Framework
Stage Stage Stage
L...
Internal structureInternal structure
10/19
Ivan Zappia, David Parlanti, Federica Paganelli / LiSEP: a Lightweight and Exte...
Use case example:
statement registration
Use case example:
statement registration
11/19
Ivan Zappia, David Parlanti, Feder...
EvaluationEvaluation
Submitted events are enqueued in
batches according to registered
statements (From Clause Manager).
Wh...
Performance analysis (1/2)Performance analysis (1/2)
14/19
Ivan Zappia, David Parlanti, Federica Paganelli / LiSEP: a Ligh...
2nd test case, high-expressivity
correlation statement:
• greater batch size leads to a
more computationally
challenging p...
Case study (1/3)Case study (1/3)
Case study on dangerous goods monitoring in maritime transport routes.
This activity is p...
Case study (2/3)Case study (2/3)
The following statement is built to detect the “possible fire” complex event
from raw tem...
Case study (3/3)Case study (3/3)
17/19
Ivan Zappia, David Parlanti, Federica Paganelli / LiSEP: a Lightweight and Extensib...
ConclusionsConclusions
Distinctive LiSEP traits:
• stage-based and modular architecture according to the SEDA pattern; thi...
Further investigationsFurther investigations
Areas that could benefit from specific further
investigations:
• the Event Pr...
LiSEP: a Lightweight and Extensible tool for
Complex Event Processing
Thanks!
Upcoming SlideShare
Loading in …5
×

A Lightweight and Extensible Complex Event Processing System for Sense and Respond Applications

704 views
604 views

Published on

Complex Event Processing (CEP) is considered as a promising technology for enabling the evolution of service-oriented enterprise systems towards operational aware systems. CEP effectively supports the implementation of "sense and respond" behaviours, as it enables to extract meaningful events from raw data streams originated by sensing infrastructures, for enterprise processes and applications consumption. This paper proposes a novel CEP engine conceived with ease of use, extensibility, portability, and scalability requirements in mind. More specifically, we propose a Lightweight Stage-based Event Processor (LiSEP) based on a layered architectural design. Thanks to the adoption of Staged Event-Driven Architecture principles, core event processing logic is decoupled from low-level thread management issues. This results in an easy-to-understand and extensible implementation while testing results show performance scalability. We report on the carrying out of a case study on dangerous goods monitoring in maritime transport. The objective of the case study is to develop a Proof of Concept application leveraging on LiSEP capabilities in sensor and RFID events processing for monitoring and alerting purposes.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
704
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
27
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

A Lightweight and Extensible Complex Event Processing System for Sense and Respond Applications

  1. 1. LiSEP: a Lightweight and Extensible tool for Complex Event Processing Ivan Zappia, David Parlanti, Federica Paganelli National Interuniversity Consortium for Telecommunications Firenze, Italy
  2. 2. ReferencesReferences 1/19 Ivan Zappia, David Parlanti, Federica Paganelli / LiSEP: a Lightweight and Extensible tool for Complex Event I. Zappia, F. Paganelli, D. Parlanti, "A Lightweight and Extensible Complex Event Processing System for Sense and Respond Applications", Expert Systems with Applications, 39 (12), pp. 10408-10419,2012, http://dx.doi.org/10.1016/j.eswa.2012.01.197. I. Zappia, D. Parlanti, F. Paganelli, "LiSEP: a Lightweight and Extensible tool for Complex Event Processing", Proceedings - 2011 IEEE International Conference on Services Computing, SCC 2011, art. no. 6009325 , pp. 701-708, Washington, 2011, http://dx.doi.org/10.1109/SCC.2011.63
  3. 3. ContextContext 1/19 Ivan Zappia, David Parlanti, Federica Paganelli / LiSEP: a Lightweight and Extensible tool for Complex Event Need for integrating heterogeneous information systems has promoted the increasing adoption of architectural patterns based on loose-coupling and message exchange. Event-Driven Architecture is based on a push asynchronous communication mechanism; messages (events) are used to propagate system state alterations. Specific analysis tools are needed to manipulate great volumes of events, non necessarily organized, in order to obtain a more aggregate and manageable view (Event Processing). Complex Event Processing to correlate (time and causality) heterogeneous event instances so to infer and manage higher abstraction levels of information.
  4. 4. Complex Event ProcessingComplex Event Processing A family of technologies concurring at the elaboration of large amount of simple data. Goal is to identify the most valuable information samples and/or infer new entities placed at higher abstraction levels. -Events are available as streams or clouds of data, each one marked with a timestamp. -Event patterns are represented through query statements modeling temporal and causality- based relations. -Received events are continuously evaluated against registered statements so to produce prompt feedback (“inversion” of the database concept). Ivan Zappia, David Parlanti, Federica Paganelli / LiSEP: a Lightweight and Extensible tool for Complex Event 3/19
  5. 5. Research goalResearch goal This paper proposes a novel Complex Event Processing engine conceived with extensibility, portability, modularity and scalability requirements in mind. Ease of use and “lightness” are also well- accepted features. 2/19 Ivan Zappia, David Parlanti, Federica Paganelli / LiSEP: a Lightweight and Extensible tool for Complex Event
  6. 6. Requirements analysisRequirements analysis Functional requirements: - system should allow event patterns definition and memorization; - system should be able to interpret a sufficiently expressive high level language designed to define said event patterns; - system should be able to accept and evaluate incoming events; - system should allow the definition of (complex) actions to be executed when a particular pattern is detected; - system should expose a public administration interface. Non-functional requirements: - portability; - modularity and extensibility; - scalability; - minimal configuration and deployment requirements. 4/19 Ivan Zappia, David Parlanti, Federica Paganelli / LiSEP: a Lightweight and Extensible tool for Complex Event
  7. 7. Event Processing Language (1/2)Event Processing Language (1/2) High level and SQL-like language; clauses have been “extended” to enable event patterns definition (event types, different kinds of constraints). SELECT o.amount AS 'OA', p.amount AS 'PA' ORDER BY 'PA' ASCENDING FROM ns.Order o WHEN 10 WITHIN '2011/02/22 08:00:00:000 CEST' AND '2011/02/22 11:30:00:000 CEST', ns.Payment p WHEN 10 STARTING_FROM '2011/01/01 08:00:00:000 CEST' EXPIRES_ON '2011/12/31 18:00:00:000 CEST' INNER_JOIN o, p ON o.id = p.orderId WHERE o.amount < 30000 5/19 Ivan Zappia, David Parlanti, Federica Paganelli / LiSEP: a Lightweight and Extensible tool for Complex Event
  8. 8. Event Processing Language (2/2)Event Processing Language (2/2) To interpret the EPL language and translate statements in objects the CEP engine can actually use we need a parser. A parser analyzes the syntactic structure of input statement strings and produces a semantically equivalent data structure, a tree in this case. As a consequence of its SQL-like syntax, this EPL language is made up of independent clauses, which are elaborated in sequence according to a pre-defined and fixed order. Adopting a divide et impera approach, five distinct parsers were designed, each of them tailored for a specific language clause. These parsers are used to compile textual clauses into clause expressions; said expressions enable the following evaluation phase. 6/19 Ivan Zappia, David Parlanti, Federica Paganelli / LiSEP: a Lightweight and Extensible tool for Complex Event
  9. 9. Design choicesDesign choices Non-functional requirements reception: • Java programming language was adopted to match the portability constraint; • an internal structure based on the Staged Event-Driven Architecture (SEDA) pattern by Matt Welsh (2001) was adopted to achieve modularity, extensibility and scalability; • the adoption of only strictly Java SE libraries (thus avoiding any application container usage) and as few as possible third party libraries (thus preventing vendor lock-ins) led to deployment requisites minimization. For these reasons our CEP tool is named Lightweight Stage- based Event Processor (LiSEP). 7/19 Ivan Zappia, David Parlanti, Federica Paganelli / LiSEP: a Lightweight and Extensible tool for Complex Event
  10. 10. Staged Event-Driven ArchitectureStaged Event-Driven Architecture 8/19 • Idea: decomposition of a complex event-driven system in stages connected by event-queues. • Main pattern goals: - efficient, event-driven concurrency (thread pools, non- blocking I/O primitives); - stages decoupling; - stages have a unique control point for incoming events; - easy load balancing (event-queues); - code modularity. Ivan Zappia, David Parlanti, Federica Paganelli / LiSEP: a Lightweight and Extensible tool for Complex Event M. Welsh, 2001
  11. 11. Layered architectureLayered architecture Hardware Operating System Java Virtual Machine SEDA Framework Stage Stage Stage LiSEP 9/19 Ivan Zappia, David Parlanti, Federica Paganelli / LiSEP: a Lightweight and Extensible tool for Complex Event
  12. 12. Internal structureInternal structure 10/19 Ivan Zappia, David Parlanti, Federica Paganelli / LiSEP: a Lightweight and Extensible tool for Complex Event
  13. 13. Use case example: statement registration Use case example: statement registration 11/19 Ivan Zappia, David Parlanti, Federica Paganelli / LiSEP: a Lightweight and Extensible tool for Complex Event
  14. 14. EvaluationEvaluation Submitted events are enqueued in batches according to registered statements (From Clause Manager). When full, each batch is forwarded to the next stage on the evaluation chain, according to an optimized, and statement-specific, Evaluation Routing Table. From there on, multiple computation paths are executed in parallel (messages asynchronicity). The Listeners Manager stage is assigned to call listeners interested in those batches that reaches this final evaluation phase. 12/19 Ivan Zappia, David Parlanti, Federica Paganelli / LiSEP: a Lightweight and Extensible tool for Complex Event
  15. 15. Performance analysis (1/2)Performance analysis (1/2) 14/19 Ivan Zappia, David Parlanti, Federica Paganelli / LiSEP: a Lightweight and Extensible tool for Complex Event • Testing machine: Intel Core i5-750 (2,72GHz) desktop with 4GB of DDR3 PC3-12800 (1600MHz) RAM. • Two test cases parameterized on variables batch size and selectivity. 1st test case, filtering statement: • greater batch size leads to fewer exchanged messages and therefore higher throughput values; • greater selectivity leads to fewer events in the latter stages and therefore higher throughput values.
  16. 16. 2nd test case, high-expressivity correlation statement: • greater batch size leads to a more computationally challenging projection phase; • configuring four threads on the Join stage, performances triplicate on the reference multi- core machine. Performance analysis (2/2)Performance analysis (2/2) 14/19 Ivan Zappia, David Parlanti, Federica Paganelli / LiSEP: a Lightweight and Extensible tool for Complex Event • Testing machine: Intel Core i5-750 (2,72GHz) desktop with 4GB of DDR3 PC3-12800 (1600MHz) RAM. • Two test cases parameterized on variables batch size and selectivity.
  17. 17. Case study (1/3)Case study (1/3) Case study on dangerous goods monitoring in maritime transport routes. This activity is part of an ongoing research project, called SITMAR (acronym for the Italian equivalent of “Integrated system for goods maritime transport in multi- modal scenarios”), funded by the Italian Ministry for Economic Development. The monitoring and control infrastructure is based on a set of RFIDs and sensors, deployed in the container (the reference loading unit) and/or in the ship storage area in order to monitor and control the environment conditions. LiSEP can be configured in order to: - deliver filtered and aggregated events to a monitoring application component deployed locally - detect anomalies in the physical parameters of containers and the storage area in order to trigger alert services 15/19 Ivan Zappia, David Parlanti, Federica Paganelli / LiSEP: a Lightweight and Extensible tool for Complex Event
  18. 18. Case study (2/3)Case study (2/3) The following statement is built to detect the “possible fire” complex event from raw temperature sensor readings and simple event LightDetected occurrences. SELECT l.senderId AS ‘senderId’, t.timestamp AS ‘T timestamp’, l.timestamp AS ‘L timestamp’, t.value AS ‘T value’ FROM model.events.LightDetected l BATCH 1 model.events.Temperature t BATCH 1 FILTER t.value >= <temperatureThreshold> INNER_JOIN t, l ON t.senderId = l.senderId WHERE t.senderId = <containerId> & (t.timestamp − l.timestamp < 10000 | l.timestamp − t.timestamp < 10000) 16/19 Ivan Zappia, David Parlanti, Federica Paganelli / LiSEP: a Lightweight and Extensible tool for Complex Event
  19. 19. Case study (3/3)Case study (3/3) 17/19 Ivan Zappia, David Parlanti, Federica Paganelli / LiSEP: a Lightweight and Extensible tool for Complex Event
  20. 20. ConclusionsConclusions Distinctive LiSEP traits: • stage-based and modular architecture according to the SEDA pattern; this approach clearly separates the core logic devoted to event processing from the inner communication mechanism and low-level thread management; • SEDA framework adoption eases system configuration, thus allowing available resources exploitation (scalability); • each stage is extremely simple and intelligible; • high maintainability and extensibility given the high separation of concerns through the autonomous and specialized units constituting the engine; adjustments can be performed with limited impact and only on strictly related areas; • deployment procedure is simple given the independence from specific libraries other than standard Java SE platform; • independence from third party libraries concurs in producing a lightweight engine, which can be easily integrated in pre-existing applications. 18/19 Ivan Zappia, David Parlanti, Federica Paganelli / LiSEP: a Lightweight and Extensible tool for Complex Event
  21. 21. Further investigationsFurther investigations Areas that could benefit from specific further investigations: • the Event Processing Language may be extended to increase system features (new negation operator, extension of present temporal operators); • the SEDA framework may be extended with a self- tuning module so to optimize exploitation of available resources (e.g., changing number of threads allocated per stage). 19/19 Ivan Zappia, David Parlanti, Federica Paganelli / LiSEP: a Lightweight and Extensible tool for Complex Event
  22. 22. LiSEP: a Lightweight and Extensible tool for Complex Event Processing Thanks!

×