• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Event Processing and Stream Reasoning with ETALIS
 

Event Processing and Stream Reasoning with ETALIS

on

  • 962 views

 

Statistics

Views

Total Views
962
Views on SlideShare
958
Embed Views
4

Actions

Likes
0
Downloads
48
Comments
0

3 Embeds 4

https://jujo00obo2o234ungd3t8qjfcjrs3o6k-a-sites-opensocial.googleusercontent.com 2
http://www.linkedin.com 1
https://www.linkedin.com 1

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Event Processing and Stream Reasoning with ETALIS Event Processing and Stream Reasoning with ETALIS Presentation Transcript

    • INFORMATIK FZI FORSCHUNGSZENTRUMEvent Processing and Stream Reasoning withETALISDarko AnicicAIFB Graduiertenkolloquium, KITAIFB Graduate Colloquium, KIT
    • Event Processing & Stream Reasoning MOTIVATION23.11.2011 © FZI Forschungszentrum Informatik 2
    • Real Time Information Processing Business Amount of Time to pressure to generated react detect digitalised relevant RT information situations Ensure high Ensure Leave more throughput timeliness time for computing and low- appropriate latency reactions Event Processing23.11.2011 © FZI Forschungszentrum Informatik 3
    • Shifting Event Processing Toward More IntelligentEvent Processing iEP EP23.11.2011 © FZI Forschungszentrum Informatik 4
    • Shifting Reasoning Toward Stream Reasoning Stream Reasoning Reasoning23.11.2011 © FZI Forschungszentrum Informatik 5
    • Scenario: Finding a Path Between Two Cities Deductive reasoning can solve the problem23.11.2011 © FZI Forschungszentrum Informatik 6
    • Scenario: Traffic Monitoring Event Processing solves the problem23.11.2011 © FZI Forschungszentrum Informatik 7
    • Scenario: Finding a Path in Real Time Stream Reasoning solves the problem23.11.2011 © FZI Forschungszentrum Informatik 8
    • Preliminaries & Related Work INTRODUCTION23.11.2011 © FZI Forschungszentrum Informatik 9
    • Event Processing An event is defined as an occurrence within a particular system or domain. It is something that has happened, or is contemplated as having happened in that domain [Etzion and Niblett, EPIA‟10]. Event Processing is computing that performs operations on events. Common event processing operations include reading, creating, transforming, and deleting events [Etzion and Niblett, EPIA‟10]. Event-driven interactions vs. request-response interactions Asynchronous interactions vs. synchronous interactions Information Push vs. Information Pull Events as a means to declare changes  the principle of decoupling With respect to Logic Programming...23.11.2011 © FZI Forschungszentrum Informatik 10
    • Overview of ETALIS Approach ETALIS Foundation ETALIS Execution EPN in Language Model ETALIS ETALIS Extensions Retraction in Out-of-Order EP-SPARQL EP EP Practical Considerations Implementation Evaluation ETALIS in Use23.11.2011 © FZI Forschungszentrum Informatik 11
    • Related Work Active Databases • XChangeEQ [Bry al.‟89] HiPAC [McCarthy et al.‟07] • Introducing time into RDF • Ode [Gehani et al.„92] rules Homogenous reaction [Gutierrez et al.‟07] • SAMOS [Gatziu et al.„92] [Paschke et al.‟07] • SPARQL-ST [Perry et al.‟07] Event Processing Systems • Snoop [Chakravarthy et Statelog [Lausen et al.‟96] TelegraphCQ • Temporal SPARQL [Tappolet • • al.„94]Maintenance System [Chandrasekaran et al.‟03] ECA rules with process Truth et al.‟09] • Amit [Adi [Behrends et al.‟96] Sentinel [Chakravarthy„97] algebras et al.‟04] [Doyle‟78 and „79] Logic-Based Approaches • stSPARQL [Koubarakis et • • Cayuga [Carney et al.‟07] SnoopIB[Demers et al.‟02] Borealis [Adaikkalavan et Incremental Reasoning on • al.‟10] stream processing with Event [BargaRich • • al.„06] and et al.‟07] Streams CEDR Replay-based revision al.‟10] • C-SPARQL [Barbieri et [Li et out-of-order data arrival Retraction in EP • Background sliding windows Semantics ofKnowledge [Maskey etKnowledge Bases • Streaming al.‟02] al.‟07] et al.‟09] • [Barbieri et streaming [Barga Consistent al.‟10] [Krämer • [Walavalkarstreaming [Barga Consistent etet al.‟10] al.‟08] • Prova [Kozlenkov et al.‟06] ZStream [Mei et al.‟02] SPARQL [Bolles • Streaming et al.‟02] et Out-of-Order EP • SASE [Wu • et al.‟08] Pattern Query Sequence al.‟06, Gyllstrom et • Incrementalover Out-of-on Processing Reasoning al.‟08, Agrawal et al.‟08] Semantic-Based Approaches Streams and Rich • Order Event Streams [Liu et Esper, Coral8, StreamSQL, Background Knowledge al.‟09] CCL etc. • [Barbieri et al.‟10] Speculative out-of-order events [Brito al.‟09]23.11.2011 © FZI Forschungszentrum Informatik 12
    • Research Questions Can we devise a uniform formalism to formally express both, complex event patterns and background knowledge for EP and SR? How to effectively use logic inference to derive complex events in a timely fashion (in an event-driven fashion)? By realising EP with concepts from LP, can we detect more real time situations that are otherwise undetectable with sole EP? Would an LP approach for EP be extensible enough for specific requirements found in EP? Do we need to compromise on performance, to get in return detections based on events patterns and background knowledge?23.11.2011 © FZI Forschungszentrum Informatik 13
    • Overview of ETALIS Approach ETALIS Foundation ETALIS Execution EPN in Language Model ETALIS ETALIS Extensions Retraction in Out-of-Order EP-SPARQL EP EP Practical Considerations Implementation Evaluation ETALIS in Use23.11.2011 © FZI Forschungszentrum Informatik 14
    • Design Principles and Requirements (I) Formal declarative semantics  Patterns describe what situations need to be detected, and do not specify possible ways of detecting them, the order of execution etc. Point-based vs. interval-based temporal semantics  Interval-based semantics enables richer semantics  possible inconsistencies encountered with point-based events, e.g., e1 before (e2 before e3) Seamless integration of events with queries  Databases are often used in enriching events with additional data  Support for query processing (including recursive queries) Seamless integration of events with domain knowledge  To enable reasoning about events and knowledge  Derivation of implicit information in order to propose recommendations, or to accomplish event classification, clustering, filtering23.11.2011 © FZI Forschungszentrum Informatik 15
    • Role of Logic in Event Processing Declarative semantics to ground well defined behaviour of event- based systems On-the-fly adaptive: everything is data (patterns can be as easy changed and adapted as data) Justifications: why did an event occur? Why didn‟t it occur? Reasoning about events (over time, space, context, their relations and constraints):  Contradicting complex events/situations;  Detection of not yet fulfilled complex patterns (e.g., 80% fulfilled event);  Event retraction (revision) and out-of-order events. Detection of complex events, states, situations of interest, and further controlling reactive behaviour (actions/reactions) triggered by detected events; Pattern rule management: consistency checking, minimal set of pattern rules, correctness of pattern rules etc.23.11.2011 © FZI Forschungszentrum Informatik 16
    • Design Principles and Requirements (II) Event-driven incremental reasoning  Events derived in timely fashion and in the asynchronous push mode Expressivity  Support for various event processing agents  Support for various other features for event-driven applications Set at a time vs. event at a time processing  Computation is performed whenever a relevant event occurs Simplicity and ease-of-use  Rules: data extraction, event composition (event hierarchies), temporal and causal relations, aggregations, non-monotonic features Extensibility  Capability to support extensions23.11.2011 © FZI Forschungszentrum Informatik 17
    • ETALIS Language for Events - SyntaxA predicate name with t is a term of q is a nonnegativearity n, t(i) denotes terms type boolean rational number BIN: SEQ, AND, PAR, OR, EQUALS, MEETS, STARTS, or FINISHES Event rule is defined as a formula of the following shape: where p is an event pattern containing all variables occurring in23.11.2011 © FZI Forschungszentrum Informatik 18
    • ETALIS: Interval-based Semantics23.11.2011 © FZI Forschungszentrum Informatik 19
    • ETALIS Language for Events - Semantics23.11.2011 © FZI Forschungszentrum Informatik 20
    • ELE: Complexity Properties Complexity Properties depend on the conditions put on the formalism‟s signature EXPTIME-complete, without further restrictions [Dantsin et al. (2001)] The formalism is decidable and tractable if both C, and the arity of functions and predicates, is bounded Function free Horn logic is hard for PTIME Function symbol “materialisation” can be done in polynomial time There are polynomially many static ground atoms There are polynomially many event ground atoms to be possible entailed Dantsin, E., Eiter, T., Gottlob, G., & Voronkov, A. (2001). Complexity and expressive power of logic programming. ACM Computing Surveys, 33(3), 374–425. 23.11.2011 © FZI Forschungszentrum Informatik 21
    • ETALIS: Operational Semantics (SEQ) 1. Complex event pattern a SEQ b SEQ c → ce1 ((a SEQ b) SEQ c) → ce1 2. Decoupling a SEQ b → ie 3. Binarization ie SEQ c → ce1 4. Event-driven backward chaining (EDBC) rules23.11.2011 © FZI Forschungszentrum Informatik 22
    • Order of Rule Execution: SEQ & AND a ie c SEQ b → a OP1 OP2 b SEQ a → ie c b c SEQ b → a b AND a → ie23.11.2011 © FZI Forschungszentrum Informatik 23
    • Properties of EDBC Rules A simple model: two-input intermediate goals (events) Goals are automatically asserted by rules as relevant events occur Goals are persisted over a period of time “waiting” to support detection of a more complex goal Goals are unique Only goals useful w.r.t the given patterns are computed Rules are executed backwards, but they exhibit a forward chaining behaviour During rule evaluation, no backtracking occurs23.11.2011 © FZI Forschungszentrum Informatik 24
    • An Example23.11.2011 © FZI Forschungszentrum Informatik 25
    • Examples of Iterative and Aggregative Patterns The k-fold sequential execution of an event a: A length-based window of size n: A sum over an unbound event stream until a threshold value is met:23.11.2011 © FZI Forschungszentrum Informatik 26
    • Event Processing Network23.11.2011 Source: O. Etzion undFZI Niblett. Event Processing in Action © P. Forschungszentrum Informatik 27
    • Event Processing Agents Event Processing Agent Pattern Filter Transformation detection Translate Aggregate Split Compose Enrich Project23.11.2011 Source: O. Etzion und P. Niblett. Event Processing in Action 28
    • Event Filtering in ETALIS Language23.11.2011 © FZI Forschungszentrum Informatik 29
    • Overview of ETALIS Approach ETALIS Foundation ETALIS Execution EPN in Language Model ETALIS ETALIS Extensions Retraction in Out-of-Order EP-SPARQL EP EP Practical Considerations Implementation Evaluation ETALIS in Use23.11.2011 © FZI Forschungszentrum Informatik 30
    • Event Retractions in ETALIS Events are often assumed to be immutable and therefore always correct In some situations however revisions are required:  an event was reported by mistake, but did not happen in reality  an event was triggered and later revoked due to a transaction failure Events used for transaction monitoring and auditing may be retracted when a transaction fails. As recognised in [Ryvkina et al. ICDE‟06], event stream sources may issue revision tuples that amend previously issued events ETALIS takes revision tuples into account and produce correct revision outputs.23.11.2011 © FZI Forschungszentrum Informatik 31
    • ETALIS: Operational Semantics (rSEQ) Standard EDBC rules for SEQ Additional EDBC rules for retraction Rules to save ie1 and enable its retraction
    • Processing Out-of-Order Events Events are often assumed to be totally ordered Delays caused by network latencies, sensor and machine failures Delayed events are known as out-of-order eventsce1 ← stock(Agent1, “GO”, Pr1,Vol1) SEQ stock(Agent2, “GO”, Pr2,Vol2) WHERE Pr1*1.20<Pr2.ce2 ← stock(Agent1, “MS”, Pr1,Vol1) SEQ stock(Agent2, “MS”, Pr2,Vol2) WHERE Pr1>1.20*Pr2. Missing complex events due to out-of-order stream: stock(agent1, “GO”, 100,10) SEQ stock(agent2, “GO”, 125,10); False positives complex event ce2 due to an out-of-order event.23.11.2011 © FZI Forschungszentrum Informatik 33
    • ETALIS: Operational Semantics (outSEQ) Additional EDBC rules for out-of-order23.11.2011 © FZI Forschungszentrum Informatik 34
    • EP-SPARQL: Toward Real-Time Semantic Web Rapidly changing data Static or slowly evolving represented as events background knowledge handles handles Event Processing Semantic Web technologies (EP) including EP SPARQL EP-SPARQL • Temporal relatedness • Semantic relatedness • Stream reasoning23.11.2011 © FZI Forschungszentrum Informatik 35
    • EP-SPARQL - Syntax Extends SPARQL to enable event-based processing by taking into account temporal situatedness of triple assertions. Syntactical and semantic downward-compatibility to plain SPARQL. Operators: FILTER, AND, UNION, OPTIONAL, SEQ, EQUALS, OPTIONALSEQ, a nd EQUALSOPTIONAL getDURATION() yields a literal of type xsd:duration giving the time interval associated to the graph pattern getSTARTTIME() and getENDTIME() retrieve the time stamps of type xsd:dateTime of the start and end of the interval;23.11.2011 © FZI Forschungszentrum Informatik 36
    • EP-SPARQL - Semantics23.11.2011 © FZI Forschungszentrum Informatik 37
    • EP-SPARQL Example: Traffic Monitoring23.11.2011 © FZI Forschungszentrum Informatik 38
    • Overview of ETALIS Approach ETALIS Foundation ETALIS Execution EPN in Language Model ETALIS ETALIS Extensions Retraction in Out-of-Order EP-SPARQL EP EP Practical Considerations Implementation Evaluation ETALIS in Use23.11.2011 © FZI Forschungszentrum Informatik 39
    • ETALIS: System Diagram Parser Compiler Execution Auxiliary components
    • EP-SPARQL: System Diagram Parser RDFS Parser and Compiler Compiler Execution
    • ETALIS Interfaces23.11.2011 © FZI Forschungszentrum Informatik 42
    • Performance Evaluation - Settings Intel Core Quad CPU Q9400 2,66GHz, 8GB of RAM running Windows Vista x64 SWI Prolog version 5.6.64 YAP Prolog version 5.1.3 Esper 3.3.0 All tested engines ran in a single dedicated CPU core Output generated from all tests is validated Data sets:  Stream generator creates time series data with probabilistic values  Streams with stock data from Google Finance and Yahoo Finance  sensor readings from the National Data Buoy Center (NDBC)  subclass relations computed with the Ethan Plants ontology  to explore routes in Milan we use the Milan ontology  GeoNames ontologies to identify important geographic locations affected by weather observations detected in our use case23.11.2011 © FZI Forschungszentrum Informatik 43
    • Common Operators I Test patterns: Esper 3.3.0 P-SWI P-YAP Esper 3.3.0 P - SWI P - YapThroughput (1000 x Events/Sec) Throughput (1000 x Events/Sec) 35 30 30 25 25 20 20 15 15 10 10 5 5 0 0 25 50 75 100 25 50 75 100 Event stream size x 1000 Event stream size x 1000 23.11.2011 © FZI Forschungszentrum Informatik 44
    • Common Operators IITest patterns: Esper 3.3.0 P-SWI P-Yap Esper 3.3.0 P-SWI P-YapThroughput (1000 x Events/Sec) Throughput (1000 x Events/Sec) 50 10 45 9 40 8 35 7 30 6 25 5 20 4 15 3 10 2 5 1 0 0 25 50 75 100 2.5 5 7.5 10 Event stream size x 1000 Event stream size x 100023.11.2011 © FZI Forschungszentrum Informatik 45
    • Iterative Event PatternsTest patterns: Supply chain Check between from paths the path 100 and 5000 the beginning or from the last event23.11.2011 © FZI Forschungszentrum Informatik 46
    • Extensions ITest patterns: Retraction in EP with ETALIS Revision Flag off Revision Flag on 35 Out of order In orderThroughput (1000 x Events/Sec) Throughput (1000 x Events/Sec) 30 50.0 25 40.0 Out-of-order 20 EP with 30.0 15 ETALIS 20.0 10 10.0 5 0 0.0 SEQ AND PAR OR 0% 10% 20% 33% Operator Percentage of out-of-order 23.11.2011 © FZI Forschungszentrum Informatik 47
    • Scenario: Stream Reasoning Evaluation  A Goods Delivery system in the city of Milan  An agent delivers goods to a certain location  While visiting a location, the system “listens” to traffic events related to the next location  Inaccessible routes are recomputed on-the-fly 1 Visitor 10 Visitors 1 Visitor 10 Visitors 1400 1600 Consumed Memory in kBConsumed time in ms 1200 1400 1000 1200 1000 800 800 600 600 400 400 200 200 0 0 5 10 15 20 5 10 15 20 Number of locations Number of locations 23.11.2011 © FZI Forschungszentrum Informatik 48
    • ETALIS in Use • Event Processing in Action, by The Fast Flower Opher Etzion and Peter Niblett Delivery in EPIA • An EPN implemented in ETALIS The Drug • Collaborative work on drug design Discovery in SYNERGY • ETALIS: extending SOA with EDA On the Live • MesoWest sensor networkMeasurements of Environmental • Analysing sensor data over time and geographical space Phenomena23.11.2011 © FZI Forschungszentrum Informatik 49
    • Conclusions and Outlook SUMMARY23.11.2011 © FZI Forschungszentrum Informatik 50
    • Overview of ETALIS Approach ETALIS Foundation RR’10, RuleML’11(I), REDS’10, AAIJ’11 ETALIS Execution EPN in Language Model ETALIS ETALIS Extensions RuleML’11(II), PADL’11, WWW’11 Retraction in Out-of-Order EP-SPARQL EP EP Practical Considerations SWJ’11 Implementation Evaluation ETALIS in Use23.11.2011 © FZI Forschungszentrum Informatik 51
    • Research Questions Can we devise a uniform formalism to formally express both, complex event patterns and background knowledge for EP and SR?  ETALIS Language for Events How to effectively use logic inference to derive complex events in a timely fashion (in an event-driven fashion)?  EDBC rules By realising EP with concepts from LP, can we detect more real time situations that are otherwise undetectable with sole EP?  The approach pays off when background knowledge is evolving Would an LP approach for EP be extensible enough for specific requirements found in EP?  Retraction in EP, out-of-order EP, EP-SPARQL Do we need to compromise on performance, to get in return detections based on events patterns and background knowledge?  It depends how big and complex background knowledge is23.11.2011 © FZI Forschungszentrum Informatik 52
    • Outlook: Event-Driven Dynamic Processes23.11.2011 © FZI Forschungszentrum Informatik 53
    • Thank you for your attention!References:Anicic et al. Real-Time Complex Event Recognition and Reasoning – Anicic et al. EP-SPARQL: A Unified Language for Event ProcessingA Logic Programming Approach, the Applied Artificial Intelligence and Stream Reasoning, WWW 2011.journal, Special Issue on Event Recognition. 2011. Anicic et al. A Rule-Based Language for Complex Event ProcessingAnicic et al. Stream Reasoning and Complex Event Processing in and Reasoning, RR 2010ETALIS, Semantic Web Journal, Special Issue: Semantic Web Toolsand Systems. 2011. Fodor, Anicic, Rudolph. Results on Out-of-Order Event Processing, PADL 2011.Anicic et al. Retractable Complex Event Processing and StreamReasoning, RuleML 2011. Xu, N.Stojanovic, Lj.Stojanovic, Anicic, Studer. An approach for more efficient energy consumption based on real-time situationalAnicic et al. ETALIS: Rule-Based Reasoning in Event awareness, ESWC 2011.Processing, Chapter in Reasoning in Event-based DistributedSystems, 2010. Springer. N.Stojanovic, Milenovic, Xu, Lj.Stojanovic, Anicic, Studer. An intelligent event-driven approach for efficient energy consumption inAnicic et al. A Declarative Framework for Matching Iterative and commercial buildings: smart office use case, DEBS 2011.Aggregative Patterns against Event Streams, RuleML 2011. Apostolou, Stojanovic, Anicic. Responsive Knowledge ManagementAnicic et al. Event-driven Approach for Logic-based Complex Event for Public Administration: an Event-Driven Approach, IEEE IntelligentProcessing, The IEEE International Conference on Computational Systems, Special Issue: Transforming E-government & E-Science and Engineering 2009. participation. 2009.23.11.2011 © FZI Forschungszentrum Informatik 54