Applying Complex Event Processing        with Drools Fusion                                Edson Tirelli                  ...
Agendao   Brief introduction on CEP and Terminologyo   Drools Visiono   Drools Fusion: Complex Event Processing extensions...
Terminology: Event“An event is an observable occurrence.”“An event in the Unified Modeling Language is a  notable occurren...
Terminology: EventFor the scope of this presentation: “An event is a significant change of   state at a particular point i...
Terminology: Complex Event    “Complex Event, is an abstraction of other events      called its members.”o   Examples:    ...
Terminology: CEP“Complex Event Processing, or CEP, is primarily  an event processing concept that deals with the  task of ...
Terminology: CEPo   Examples:    o   Emergency Response Systems    o   Credit Card Fraud Detection    o   Logistics Real-T...
Terminology: CEP vs ESP    Complex Event Processing, or CEP, and Event     Stream Processing, or ESP, are two     technolo...
Terminology: CEP and ESPFor the scope of this presentation:    “CEP is used as a common term      meaning both CEP and ESP.”
Terminology: EDA“Event Driven Architecture (EDA) is a software architecture  pattern promoting the production, detection, ...
EDA vs CEP                    CEP is a component of the EDASource: http://elementallinks.typepad.com/.shared/image.html?/p...
EDA vs SOAo   EDA is **not** SOA 2.0o   Complementary architectureso   Metaphor     o In our body:         o SOA is used t...
EDA vs SOASource: http://soa-eda.blogspot.com/2006/11/how-eda-extends-soa-and-why-it-is.html
Complex Event Processingo   A few characteristics of common CEP scenarios:     o Huge volume of events, but only a few of ...
Drools Vision“A common platform to model and govern the     business logic of the enterprise.”
Drools Fusion: Enables…•   Event Detection:     • From an event cloud or set of streams, select all the       meaningful e...
Drools Fusiono   Features:     o Event Semantics as First Class Citizens     o Allow Detection, Correlation and Compositio...
Demoo   Twitter Stream CEP Demo:     o Listen to the Twitter Stream API         Twitter4J API         Listens to a rando...
Event Declaration and Semantics// declaring existing classimport some.package.VoiceCall                                o  ...
Temporal Reasoningo   Semantics for:    o time: discrete    o events: point-in-time and intervalo   Ability to express tem...
Temporal Relationshipsrule “Shipment not picked up in time”when  Shipment( $pickupTime : scheduledPickupTime )  not Shipme...
Temporal Relationshipsrule “Shipment not picked up in time”when  Shipment( $pickupTime : scheduledPickupTime )  not Shipme...
Allen’s 13 Temporal Operators                    Point-Point   Point-Interval   Interval-Interval                A A befor...
Allen’s 13 Temporal Operators                      Point-Point   Point-Interval   Interval-Interval                  A   A...
Streams : Simple Example Scenario
Stream Support (entry-points)o   A scoping abstraction for stream support     o Rule compiler gather all entry-point decla...
Cloud Mode, Stream Mode, Session Clock             CLOUD                             STREAM•   No notion of “flow of time”...
Reference Clocko   Reference clock defines the flow of timeo   Named Session Clock    o is assigned to each session create...
Session Clocko Uses the strategy pattern and multiple implementations:   o Real-time operation   o Tests   o Simulations  ...
Session Clocko Selecting the session clock:   o API:      KnowledgeSessionConfiguration conf = ...      conf.setOption( Cl...
Sliding Window Supporto   Allows reasoning over a moving window of “interest”     o Time     o Length                     ...
Sliding Window Supporto   Allows reasoning over a moving window of “interest”     o Time     o Length                     ...
Sliding Window Supporto   Allows reasoning over a moving window of “interest”     o Time     o Lengthrule “Average Order V...
Delaying Ruleso Negative patterns may require rule firings to be delayed.rule “Order timeout”rule “Order timeout”whenwhen ...
Delaying Ruleso Negative patterns may require rule firings to be delayed.rule “Order timeout”rule “Order timeout”whenwhen ...
Temporal Dimensiono   Requires the support to the temporal dimension    o A rule/query might match in a given point in tim...
Temporal Dimension Supporto   CEP scenarios are stateful by nature.o   Events usually are only interesting during a short ...
Simple Example Rulerule “Bag was lost”rule “Bag was lost”whenwhen      $b : BagScannedEvent() from entry-point “check-in” ...
Abstract Example Rulerule “Abstract event relationship example”rule “Abstract event relationship example”whenwhen      $a ...
Temporal Dependency Matrix        [-2,2]          B          [1,2]                                                        ...
Q&Ao   Drools project site:     o http://www.drools.org ( http://www.jboss.org/drools/ )o   Documentation:     o http://ww...
Applying CEP Drools Fusion - Drools jBPM Bootcamps 2011
Upcoming SlideShare
Loading in...5
×

Applying CEP Drools Fusion - Drools jBPM Bootcamps 2011

6,354

Published on

Published in: Technology, Business

Applying CEP Drools Fusion - Drools jBPM Bootcamps 2011

  1. 1. Applying Complex Event Processing with Drools Fusion Edson Tirelli etirelli@redhat.com Lead CEP Engineer JBoss, a Division of Red Hat
  2. 2. Agendao Brief introduction on CEP and Terminologyo Drools Visiono Drools Fusion: Complex Event Processing extensions o Event Declaration and Semantics o Event Cloud, Streams and the Session Clock o Temporal Reasoning o Sliding Window Support o Streams Support o Memory Managemento Questions & Answers
  3. 3. Terminology: Event“An event is an observable occurrence.”“An event in the Unified Modeling Language is a notable occurrence at a particular point in time.” http://www.wikipedia.org“Anything that happens, or is contemplated as happening.”“An object that represents, encodes or records an event, generally for the purpose of computer processing” http://complexevents.com
  4. 4. Terminology: EventFor the scope of this presentation: “An event is a significant change of state at a particular point in time”
  5. 5. Terminology: Complex Event “Complex Event, is an abstraction of other events called its members.”o Examples: o The 1929 stock market crash – an abstraction denoting many thousands of member events, including individual stock trades) o The 2004 Indonesian Tsunami – an abstraction of many natural events o A completed stock purchase -an abstraction of the events in a transaction to purchase the stock o A successful on-line shopping cart checkout – an abstraction of shopping cart events on an on-line website  Source: http://complexevents.com
  6. 6. Terminology: CEP“Complex Event Processing, or CEP, is primarily an event processing concept that deals with the task of processing multiple events with the goal of identifying the meaningful events within the event cloud.CEP employs techniques such as detection of complex patterns of many events, event correlation and abstraction, event hierarchies, and relationships between events such as causality, membership, and timing, and event- driven processes.” -- wikipedia
  7. 7. Terminology: CEPo Examples: o Emergency Response Systems o Credit Card Fraud Detection o Logistics Real-Time Awareness solution o Neonatal ICU: infant vital signs monitoring
  8. 8. Terminology: CEP vs ESP Complex Event Processing, or CEP, and Event Stream Processing, or ESP, are two technologies that were born separate, but converged.● An oversimplification: In their origins...  Event Stream Processing focused on the ability to process high volume streams of events.  Complex Event Processing focused on defining, detecting and processing the relationships among events.
  9. 9. Terminology: CEP and ESPFor the scope of this presentation: “CEP is used as a common term meaning both CEP and ESP.”
  10. 10. Terminology: EDA“Event Driven Architecture (EDA) is a software architecture pattern promoting the production, detection, consumption of, and reaction to events. An event can be defined as "a significant change in state"[1]. For example, when a consumer purchases a car, the cars state changes from "for sale" to "sold". A car dealers system architecture may treat this state change as an event to be produced, published, detected and consumed by various applications within the architecture.” http://en.wikipedia.org/wiki/Event_Driven_Architecture
  11. 11. EDA vs CEP CEP is a component of the EDASource: http://elementallinks.typepad.com/.shared/image.html?/photos/uncategorized/simple_event_flow.gif
  12. 12. EDA vs SOAo EDA is **not** SOA 2.0o Complementary architectureso Metaphor o In our body: o SOA is used to build our muscles and organs o EDA is used to build our sensory system
  13. 13. EDA vs SOASource: http://soa-eda.blogspot.com/2006/11/how-eda-extends-soa-and-why-it-is.html
  14. 14. Complex Event Processingo A few characteristics of common CEP scenarios: o Huge volume of events, but only a few of real interest o Usually events are immutable o Usually queries/rules have to run in reactive mode o Strong temporal relationships between events o Individual events are usually not important o The composition and aggregation of events is important
  15. 15. Drools Vision“A common platform to model and govern the business logic of the enterprise.”
  16. 16. Drools Fusion: Enables…• Event Detection: • From an event cloud or set of streams, select all the meaningful events, and only them.• [Temporal] Event Correlation: • Ability to correlate events and facts declaring both temporal and non-temporal constraints between them. • Ability to reason over event aggregation• Event Abstraction: • Ability to compose complex events from atomic events AND reason over them
  17. 17. Drools Fusiono Features: o Event Semantics as First Class Citizens o Allow Detection, Correlation and Composition o Temporal Constraints o Session Clock o Stream Processing o Sliding Windows o CEP volumes (scalability) o (Re)Active Rules o Data Loaders for Input
  18. 18. Demoo Twitter Stream CEP Demo: o Listen to the Twitter Stream API  Twitter4J API  Listens to a random sample of tweets o Detects patterns and reacts  Drools Fusion o Simple one process (multi-thread) demo  Focus on specific features
  19. 19. Event Declaration and Semantics// declaring existing classimport some.package.VoiceCall o Event semantics:declare VoiceCall o Point-in-time and Interval @role( event ) @timestamp( calltime ) @duration( duration )end o An event is a fact with a few special characteristics:// generating an event class o Usually immutable, but notdeclare StockTick @role( event ) enforced o Strong temporal relationships symbol : String price : double o Lifecycle may be managedend o Allow use of sliding windows o “All events are facts, but not all facts are events.”
  20. 20. Temporal Reasoningo Semantics for: o time: discrete o events: point-in-time and intervalo Ability to express temporal relationships: o Allen’s 13 temporal operatorso James F. Allen defined the 13 possible temporal relations between two events.o Eiko Yoneki and Jean Bacon defined a unified semantics for event correlation over time and space.
  21. 21. Temporal Relationshipsrule “Shipment not picked up in time”when Shipment( $pickupTime : scheduledPickupTime ) not ShipmentPickup( this before $pickupTime )then // shipment not picked up... action required.end
  22. 22. Temporal Relationshipsrule “Shipment not picked up in time”when Shipment( $pickupTime : scheduledPickupTime ) not ShipmentPickup( this before $pickupTime )then // shipment not picked up... Action required.end Temporal Relationship
  23. 23. Allen’s 13 Temporal Operators Point-Point Point-Interval Interval-Interval A A before B B A A meets B B AA overlaps B B AA finishes B B AA includes B B A A starts B B AA coincides B B
  24. 24. Allen’s 13 Temporal Operators Point-Point Point-Interval Interval-Interval A A after B B A A metBy B B AA overlapedBy B B AA finishedBy B B A A during B B A A finishes B B
  25. 25. Streams : Simple Example Scenario
  26. 26. Stream Support (entry-points)o A scoping abstraction for stream support o Rule compiler gather all entry-point declarations and expose them through the session API o Engine manages all the scoping and synchronization behind the scenes.rule “Stock Trade Correlation”rule “Stock Trade Correlation”whenwhen $c : Customer( type == “VIP” ) $c : Customer( type == “VIP” ) BuyOrderEvent( customer == $c, $id : id ) from entry-point “Home Broker Stream” BuyOrderEvent( customer == $c, $id : id ) from entry-point “Home Broker Stream” BuyAckEvent( sourceEvent == $id ) from entry-point “Stock Trader Stream” BuyAckEvent( sourceEvent == $id ) from entry-point “Stock Trader Stream”thenthen // take some action // take some actionendend
  27. 27. Cloud Mode, Stream Mode, Session Clock CLOUD STREAM• No notion of “flow of time”: • Notion of “flow of time”: the engine sees all facts concept of “now” without regard to time • Session Clock has an active• No attached Session Clock role synchronizing the• No requirements on event reasoning ordering • Event Streams must be• No automatic event lifecycle ordered management • Automatic event lifecycle• No sliding window support management • Sliding window support • Automatic rule delaying on absence of facts
  28. 28. Reference Clocko Reference clock defines the flow of timeo Named Session Clock o is assigned to each session createdo Synchronizes time sensitive operations o duration rules o event streams o process timers o sliding windows
  29. 29. Session Clocko Uses the strategy pattern and multiple implementations: o Real-time operation o Tests o Simulations o etc
  30. 30. Session Clocko Selecting the session clock: o API: KnowledgeSessionConfiguration conf = ... conf.setOption( ClockTypeOption.get( “realtime” ) ); o System Property or Configuration File: drools.clockType = pseudo
  31. 31. Sliding Window Supporto Allows reasoning over a moving window of “interest” o Time o Length Sliding window 1 Sliding window 2
  32. 32. Sliding Window Supporto Allows reasoning over a moving window of “interest” o Time o Length Sliding window 1 Sliding window 2 Joined window
  33. 33. Sliding Window Supporto Allows reasoning over a moving window of “interest” o Time o Lengthrule “Average Order Value over 12 hours”rule “Average Order Value over 12 hours”whenwhen $c : Customer() $c : Customer() $a : Number() from accumulate ( $a : Number() from accumulate ( BuyOrder( customer == $c, $p : price ) BuyOrder( customer == $c, $p : price ) over window:time( 12h ), over window:time( 12h ), average( $p ) ) average( $p ) )thenthen // do something // do somethingendend
  34. 34. Delaying Ruleso Negative patterns may require rule firings to be delayed.rule “Order timeout”rule “Order timeout”whenwhen $bse : BuyShares ( $id : id ) $bse : BuyShares ( $id : id ) not BuySharesAck( id == $id, this after[0s,30s] $bse ) not BuySharesAck( id == $id, this after[0s,30s] $bse )thenthen // Buy order was not acknowledged. Cancel operation // Buy order was not acknowledged. Cancel operation // by timeout. // by timeout.endend
  35. 35. Delaying Ruleso Negative patterns may require rule firings to be delayed.rule “Order timeout”rule “Order timeout”whenwhen $bse : BuyShares ( $id : id ) $bse : BuyShares ( $id : id ) not BuySharesAck( id == $id, this after[0s,30s] $bse ) not BuySharesAck( id == $id, this after[0s,30s] $bse )thenthen // Buy order was not acknowledged. Cancel operation // Buy order was not acknowledged. Cancel operation // by timeout. // by timeout.endend Forces the rule to wait for 30 seconds before firing, because the acknowledgement may arrive at any time!
  36. 36. Temporal Dimensiono Requires the support to the temporal dimension o A rule/query might match in a given point in time, and not match in the subsequent point in timeo That is the single most difficult requirement to support in a way that the engine: o stays deterministic o stays a high-performance engineo Achieved mostly by compile time optimizations that enable: o constraint tightening o match space narrowing o memory management
  37. 37. Temporal Dimension Supporto CEP scenarios are stateful by nature.o Events usually are only interesting during a short period of time.o Hard for applications to know when events are not necessary anymore o Temporal constraints and sliding windows describe such “window of interest”
  38. 38. Simple Example Rulerule “Bag was lost”rule “Bag was lost”whenwhen $b : BagScannedEvent() from entry-point “check-in” $b : BagScannedEvent() from entry-point “check-in” not BagScannedEvent( id == $b.id, this after[0s,5m] $b ) not BagScannedEvent( id == $b.id, this after[0s,5m] $b ) from entry-point “pre-load” from entry-point “pre-load”thenthen // Bag was lost, do something // Bag was lost, do somethingendend Easy to “see” that the only temporal relationship between the events defines a 5 minutes interest window.
  39. 39. Abstract Example Rulerule “Abstract event relationship example”rule “Abstract event relationship example”whenwhen $a : A() $a : A() $b : B( this after[-2, 2] $a ) $b : B( this after[-2, 2] $a ) $c : C( this after[-3, 4] $a ) $c : C( this after[-3, 4] $a ) $d : D( this after[ 1, 2] $b, this after[2,3] $c) $d : D( this after[ 1, 2] $b, this after[2,3] $c) not not E( this after[ 1,10] $d ) E( this after[ 1,10] $d )thenthen // Bag was lost, do something // Bag was lost, do somethingendend How about now? What is the temporal relationship between A and E?
  40. 40. Temporal Dependency Matrix [-2,2] B [1,2] [1,10] A D E [-3,4] C [2,3] Constraint tightening A B C D EA [ 0, 0 ] [ -2, 2 ] [ -3, 2 ] [ -1, 4 ] [ 0, 14 ]B [ -2, 2 ] [ 0, 0 ] [ -2, 0 ] [ 1, 2 ] [ 2, 12 ]C [ -2, 3 ] [ 0, 2 ] [ 0, 0 ] [ 2, 3 ] [ 3, 13 ]D [ -4, 1 ] [ -2, -1 ] [ -3, -2 ] [ 0, 0 ] [ 1, 10 ]E [ -14, 0 ] [ -12, -2 ] [ -13, -3 ] [-10,-1 ] [ 0, 0 ]
  41. 41. Q&Ao Drools project site: o http://www.drools.org ( http://www.jboss.org/drools/ )o Documentation: o http://www.jboss.org/drools/documentation.html Edson Tirelli – etirelli@redhat.com
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×