Mark Proctor Project LeadThe SkyNet funding bill is passed.The system goes online on August 4th, 1997.Human decisions are removed from strategic defense.SkyNet begins to learn at a geometric rate.It becomes self-aware at 2:14am Eastern time, August 29thIn a panic, they try to pull the plug.And, Skynet fights back
3What is C omplex E vent Proces s ing? My answer: the ability to make more money than your competitors. ;)
4What is C omplex E vent Proces s ing 1. Detect 2. Correlate 3. React
5Time is Money Business Event Value Loss Business Value Reaction Time Time Loss Adapted from a presentation by James Taylor, Sep/2011
6 Terminology: EventF or th e s cop e of th is p re s e ntation:“An e ve nt is a s ignificant change of s tate at a p articu lar point in time”
7 Terminology: Complex Event “C omplex E vent, is an ab s traction of oth e r e ve nts calle d its m e m b e rs .”E xam p le s :The 1929 s toc k market cras h – an ab s traction d e noting m any th ou s and sof m e m b e r e ve nts , inclu d ing ind ivid u al s tock trad e s )The 2004 Indones ian Ts unami – an ab s traction of m any natu ral e ve ntsA c ompleted s tock purc has e -an ab s traction of th e e ve nts in atrans action to p u rch as e th e s tockA s uc c es s ful on-line s hopping c art c hec kout – an ab s traction ofs h op p ing cart e ve nts on an on-line we b s ite S ource : http :/com p l xe ve nts.com / e
8 Terminology: CEP“C omplex E vent Proces s ing, or C E P , is p rim arily an e ve nt p roce s s ing conce p t th at d e als with th e tas k of p roce s s ing m u ltip le e ve nts with th e goal of identifying the meaningful events with in th e e ve nt clou d .C E P e m p loys te ch niqu e s s u ch as detection of com p le x p atte rns of m any e ve nts , e ve nt correlation and abs traction, e ve nt h ie rarch ie s , and re lations h ip s b e twe e n e ve nts s u ch as cau s ality, m e m b e rs h ip , and tim ing, and e ve nt- d rive n p roce s s e s .” -- wikip e d ia
9 Terminology: CEPE xam p le s :E m e rge ncy R e s p ons e S ys te m sC re d it C ard F rau d D e te ctionLogis tics R e al-Tim e Aware ne s s s olu tionN e onatal IC U : infant vital s igns m onitoring
10 Terminology: CEP vs ESP C omplex E vent Proces s ing, or C E P , and E vent S tream Proces s ing, or E S P , are two te ch nologie s th at we re b orn s e p arate , b u t conve rge d .An ove rsim p l ification: In th e ir origins ...E ve nt S tre am P roce s s ing focu s e d on th e ab ility to p roce s s h ighvolu m e s treams of e ve nts .C om p le x E ve nt P roce s s ing focu s e d on d e fining, d e te cting andp roce s s ing th e relations hips am ong e ve nts .
11 Terminology: CEP and ESPF or th e s cop e of th is p re s e ntation: “C E P is u s e d as a com m on te rm m e aning b oth C E P and E S P.”
12 Terminology: EDA“E vent Driven A rc hitecture (E DA ) is a s oftware arch ite ctu re p atte rn p rom oting th e p rod u ction, d e te ction, cons u m p tion of, and re action to e ve nts . An e ve nt can b e d e fine d as "a s ignificant ch ange in s tate "[1 ]. F or e xam p le , wh e n a cons u m e r p u rch as e s a car, th e cars s tate ch ange s from "for s ale " to "s old ". A car d e ale rs s ys te m arch ite ctu re m ay tre at th is s tate ch ange as an e ve nt to b e p rod u ce d , p u b lis h e d , d e te cte d and cons u m e d b y variou s ap p lications with in th e arch ite ctu re .” h ttp :/ e n.wikip e d ia.org/ / wiki/ ve nt_ D rive n_ Arch ite ctu re E
13 EDA vs CEP CEP is a component of the EDASource: http://elementallinks.typepad.com/.shared/image.html?/photos/uncategorized/simple_event_flow.gif
14 E DA vs S OAE D A is **not** S O A 2.0C om p le m e ntary arch ite ctu re sM e tap h orIn ou r b od y:S O A is u s e d to b u ild ou r m u s cle s and organsE D A is u s e d to b u ild ou r s e ns ory s ys te m
16 E DA vs S OASource: http://soa-eda.blogspot.com/2006/11/how-eda-extends-soa-and-why-it-is.html
17 C omplex E vent Proces s ingA few characteris tics of common C E P s cenarios :H u ge volu m e of e ve nts , b u t only a fe w of re al inte re s tU s u ally e ve nts are im m u tab leU s u ally qu e rie s / le s h ave to ru n in re active m od e ruS trong te m p oral re lations h ip s b e twe e n e ve ntsInd ivid u al e ve nts are u s u ally not im p ortantTh e com p os ition and aggre gation of e ve nts is im p ortant
21 Drools Fus ion: E nables …E vent Detection:F rom an e ve nt clou d or s e t of s tre am s , s e le ct all th e m e aningfu le ve nts , and only th e m .[Temporal] E vent C orrelation:Ab ility to corre late e ve nts and facts d e claring b oth te m p oral andnon-te m p oral cons traints b e twe e n th e m .Ab ility to re as on ove r e ve nt aggre gationE vent A bs traction:Ab ility to com p os e com p le x e ve nts from atom ic e ve nts AN D re as onove r th e m
22 Drools Fus ionFeatures :E ve nt S e m antics as F irs t C las s C itize nsAllow D e te ction, C orre lation and C om p os itionTe m p oral C ons traintsS e s s ion C lockS tre am P roce s s ingS lid ing Wind owsC E P volu m e s (s calab ility)(R e )Active R u le sD ata Load e rs for Inp u t
23 DemoTwitter S tream C E P Demo:Lis te n to th e Twitte r S tre am AP I Twitte r4J AP I Lis te ns to a rand om s am p le of twe e ts D e te cts p atte rns and re acts D rools F u s ion S im p le one p roce s s (m u lti-th re ad ) d e m o F ocu s on s p e cific fe atu re s
24 E vent Declaration and S emantics// declaring existing classimport some.package.VoiceCall E ve nt s e m antics :declare VoiceCall @role( event ) P oint-in-tim e and Inte rval @timestamp( calltime ) @duration( duration )end An e ve nt is a fact with a fe w s p e cial// generating an event class ch aracte ris tics :declare StockTick U s u ally im m u tab le , b u t not e nforce d @role( event ) S trong te m p oral re lations h ip s symbol : String price : double Life cycle m ay b e m anage dend Allow u s e of s lid ing wind ows “All e ve nts are facts , b u t not all facts are e ve nts .”
25 Temporal Reas oningS e m antics for:time: d is cre teevents : p oint-in-tim e and inte rvalAb ility to e xp re s s te m p oral re lations h ip s :Alle n’s 1 3 te m p oral op e ratorsJ ames F. A llen d e fine d th e 1 3 p os s ib le te m p oral re lationsb e twe e n two e ve nts .E iko Yoneki and J ean B acon d e fine d a u nifie d s e m antics fore ve nt corre lation ove r tim e and s p ace .
26 Temporal Relations hipsrule “Shipment not picked up in time”when Shipment( $pickupTime : scheduledPickupTime ) not ShipmentPickup( this before $pickupTime )then // shipment not picked up... action required.end
27 Temporal Relations hipsrule “Shipment not picked up in time”when Shipment( $pickupTime : scheduledPickupTime ) not ShipmentPickup( this before $pickupTime )then // shipment not picked up... Action required.end Temporal Relationship
28 A llen’s 13 Temporal Operators Point-Point Point-Interval Interval-Interval A A before B B A A meets B B AA overlaps B B AA finishes B B AA includes B B A A starts B B AA coincides B B
29 A llen’s 13 Temporal Operators Point-Point Point-Interval Interval-Interval A A after B B A A metBy B B AA overlapedBy B B AA finishedBy B B A A during B B A A finishes B B
31 S tream S upport (entry-points )A s coping abs traction for s tream s upportR u le com p ile r gath e r all e ntry-p oint d e clarations and e xp os e th e mth rou gh th e s e s s ion AP IE ngine m anage s all th e s cop ing and s ynch ronization b e h ind th es ce ne s .rule “Stock Trade Correlation”when $c : Customer( type == “VIP” ) BuyOrderEvent( customer == $c, $id : id ) from entry-point “Home Broker Stream” BuyAckEvent( sourceEvent == $id ) from entry-point “Stock Trader Stream”then // take some actionend
32 Cloud Mode, Stream Mode, Session Clock CLOUD STREAMN o notion of “flow of tim e ”: th e N otion of “flow of tim e ”:e ngine s e e s all facts with ou t conce p t of “now”re gard to tim e S e s s ion C lock h as an activeN o attach e d S e s s ion C lock role s ynch ronizing th eN o re qu ire m e nts on e ve nt re as oningord e ring E ve nt S tre am s m u s t b eN o au tom atic e ve nt life cycle ord e re dm anage m e nt Au tom atic e ve nt life cycleN o s lid ing wind ow s u p p ort m anage m e nt S lid ing wind ow s u p p ort Au tom atic ru le d e laying on ab s e nce of facts
33 Reference C lockR e fe re nce clock d e fine s th e flow of tim eN am e d S es s ion C lockis as s igne d to e ach s e s s ion cre ate dS ynch ronize s tim e s e ns itive op e rationsd u ration ru le se ve nt s tre am sp roce s s tim e rss lid ing wind ows
34 S es s ion C lockUses the strategy pattern and multiple implementations:Real-time operationTestsSimulationsetc
35 S es s ion C lockSelecting the session clock:API: KnowledgeSessionConfiguration conf = ... conf.setOption( ClockTypeOption.get( “realtime” ) );System Property or Configuration File: drools.clockType = pseudo
36 S liding Window S upportAllows re as oning ove r a m oving wind ow of “inte re s t”Tim eLe ngth Sliding window 1 Sliding window 2
37 S liding Window S upportAllows re as oning ove r a m oving wind ow of “inte re s t”Tim eLe ngth Sliding window 1 Sliding window 2 Joined window
38 S liding Window S upportAllows re as oning ove r a m oving wind ow of “inte re s t”Tim eLe ngthrule “Average Order Value over 12 hours”when $c : Customer() $a : Number() from accumulate ( BuyOrder( customer == $c, $p : price ) over window:time( 12h ), average( $p ) )then // do somethingend
39 Delaying RulesNegative patterns may require rule firings to be delayed. rule “Order timeout” when $bse : BuyShares ( $id : id ) not BuySharesAck( id == $id, this after[0s,30s] $bse ) then // Buy order was not acknowledged. Cancel operation // by timeout. end
40 Delaying RulesNegative patterns may require rule firings to be delayed. rule “Order timeout” when $bse : BuyShares ( $id : id ) not BuySharesAck( id == $id, this after[0s,30s] $bse ) then // Buy order was not acknowledged. Cancel operation // by timeout. end Forces the rule to wait for 30 seconds before firing, because the acknowledgement may arrive at any time!
41 Temporal Dimens ionR e qu ire s th e s u p p ort to th e te m p oral d im e ns ionA ru le / e ry m igh t m atch in a give n p oint in tim e , and not m atch in quth e s u b s e qu e nt p oint in tim eTh at is th e s ingle m os t d ifficu lt re qu ire m e nt to s u p p ort in a wayth at th e e ngine :s tays d e te rm inis tics tays a h igh -p e rform ance e ngineAch ie ve d m os tly b y com p ile tim e op tim izations th at e nab le :cons traint tigh te ningm atch s p ace narrowingm e m ory m anage m e nt
42 Temporal Dimens ion S upportC E P s ce narios are s tate fu l b y natu re .E ve nts u s u ally are only inte re s ting d u ring a s h ort p e riod oftim e .H ard for ap p lications to know wh e n e ve nts are not ne ce s s aryanym oreTe m p oral cons traints and s lid ing wind ows d e s crib e s u ch “wind ow ofinte re s t”
43 S imple E xample Rulerule “Bag was lost”when $b : BagScannedEvent() from entry-point “check-in” not BagScannedEvent( id == $b.id, this after[0s,5m] $b ) from entry-point “pre-load”then // Bag was lost, do somethingendEasy to “see” that the only temporal relationship between the events defines a 5 minutes interest window.
44 C alendarsrule "weekdays are high priority" calendars "weekday" timer (int:0 1h) Execute now and after 1 hour durationwhen Alarm()then send( "priority high - we have an alarm” );endrule "weekend are low priority" calendars "weekend" timer (int:0 4h) Execute now and after 4 hour durationwhen Alarm()then send( "priority low - we have an alarm” );end
45 TimersField Name Mandatory? Allowed Values Allowed Special CharactersSeconds YES 0-59 ,-*/Minutes YES 0-59 ,-*/Hours YES 0-23 ,-*/Day of month YES 1-31 ,-*?/LWMonth YES 1-12 or JAN-DEC ,-*/Day of week YES 1-7 or SUN-SAT ,-*?/L#Year NO empty, 1970-2099 ,-*/ Send alert every quarter of an hourrule “name” timer ( cron: 0 0/15 * * * * )when Alarm( )then sendEmail( ”Alert Alert Alert!!!” )
46 A bs tract E xample Rule rule “Abstract event relationship example” when $a : A() $b : B( this after[-2, 2] $a ) $c : C( this after[-3, 4] $a ) $d : D( this after[ 1, 2] $b, this after[2,3] $c) not E( this after[ 1,10] $d ) then // Bag was lost, do something endHow about now? What is the temporal relationship between A and E?
48C E P A pplied at FedE x C us tom C ritical * Presented by Adam Mollemkopf at ORF 2009
49C E P A pplied at FedE x C us tom C ritical * Presented by Adam Mollemkopf at ORF 2009
50 C E P A pplied at FedE x C us tom C ritical * Presented by Adam Mollemkopf at ORF 2009At least 50% of Alerts can be reasoned automatically, promotingstaff savings and improved Customer and Driver experiences.Risk Avoidance via pro-active monitoringReduction in insurance claims and shipment service failuresMinimum 30% efficiency gains in shipment monitoring , saving atleast 15% of Operations staff cost.
51 C E P A pplied at FedE x C us tom C ritical * Presented by Adam Mollemkopf at ORF 2009Some numbers (from early 2010):24 x 7 sessions, no downtimeAverage of 500k+ facts/events concurrently in memoryBusiness hours: 1M+ facts/events concurrentlyResponse time for reasoning cycles:Average: 150 msPeak: 1.2 secSeveral hundred rules
52Differential Update Differential Update (a.k.a. “true modify”) Implements a real “modify/update” operation, instead of retract+assert. Reuses tuples, reduces GC stress, improves performance
Q&ADrools project s ite:h ttp :/ www.d rools .org ( h ttp :/ www.j os s .org/ rools / ) / / b dDoc umentation:h ttp :/ www.j os s .org/ rools / ocu m e ntation.h tm l / b d d Edson Tirelli – firstname.lastname@example.org