Your SlideShare is downloading. ×
0
CEP – complex event processing <ul><li>OSDC, Brisbane, November 2009 </li></ul>Michael Neale JBoss/Red Hat R&D
Michael Neale <ul><li>R&D w. JBoss (specialise in drools, now “cloud research”) </li></ul><ul><li>Open source history (use...
Outline <ul><li>Define CEP?  </li></ul><ul><li>Where is it used  </li></ul><ul><li>Events and patterns  </li></ul><ul><li>...
Definition <ul><li>Treat “inputs” as “events” (time based) </li></ul><ul><li>Look for patterns/correlations to extract som...
eg <ul><li>Event: [Hand In Air] </li></ul><ul><li>Pattern: Lots of [Hand In Air] (possibly waving, as if they didn't care)...
Huge number of events, but only a few of real interest Usually events are immutable Usually queries/rules have to run in r...
Where is it used?  <ul><li>Logistics: Package en-route tracking/warning </li></ul><ul><li>Stock market events/trades </li>...
Events and patterns <ul><li>Events immutable** facts (objects/records) </li></ul><ul><ul><li>Eg a reading from a sensor, a...
Event instance of StockTick
Patterns <ul><li>Are used to detect specific events (using constraints and higher order logic) </li></ul><ul><li>Are used ...
Sliding time windows <ul><li>Declare what “window” of time you care about </li></ul><ul><li>Needed to narrow down from hug...
Temporal reasoning <ul><li>A generalisation of “looking for co-incidences” of interest </li></ul><ul><li>Eg Hangover often...
<ul><li>Event and Time semantics: </li></ul><ul><ul><li>Point in Time </li></ul></ul><ul><ul><li>Interval-based </li></ul>...
Event semantics <ul><li>JF Allen defined a set of Time Event semantics:  </li></ul>
Temporal operators <ul><li>After </li></ul><ul><li>Before </li></ul><ul><li>Coincides </li></ul><ul><li>During </li></ul><...
Defining events <ul><li>Generally classes defined in host language (eg java, C#) </li></ul><ul><li>Some extra metadata nee...
Defining events // declaring existing class import  some.package.VoiceCall declare  VoiceCall @role ( event ) @timestamp (...
Drools Fusion <ul><li>jboss.org/drools </li></ul><ul><li>Submodule of Drools project </li></ul><ul><li>Uses rule language ...
<ul><li>rule &quot;Sound the alarm in case temperature rises above threshold&quot; </li></ul><ul><li>when </li></ul><ul><l...
Esper <ul><li>esper.codehaus.org </li></ul><ul><li>Takes “temporal SQL” approach </li></ul><ul><li>Return results for matc...
<ul><li>select fraud.accountNumber as accntNum, fraud.warning as warn,  withdraw.amount as amount, </li></ul><ul><li>MAX(f...
Deployment <ul><li>Usually a dedicated server/service endpoint </li></ul><ul><ul><li>(but could be embedded in your app if...
Concurrency... <ul><li>Need for “realtime” performance  </li></ul><ul><li>Need to make use of threads/processes </li></ul>...
FedEx example <ul><li>Presented at JavaOne, Adobe MAX etc.. </li></ul><ul><li>FedEx Custom Critical: </li></ul><ul><li>1: ...
 
 
 
 
A flex front end
 
Thanks, Q&A
Upcoming SlideShare
Loading in...5
×

Osdc Complex Event Processing

1,116

Published on

A presentation I did at OSDC in brisbane, 2009, on complex event processing. Includes a case study.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,116
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
38
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Osdc Complex Event Processing"

  1. 1. CEP – complex event processing <ul><li>OSDC, Brisbane, November 2009 </li></ul>Michael Neale JBoss/Red Hat R&D
  2. 2. Michael Neale <ul><li>R&D w. JBoss (specialise in drools, now “cloud research”) </li></ul><ul><li>Open source history (user -> fulltime developer “acquired” by jboss 2005). </li></ul><ul><li>Thanks OSDC organisers ! </li></ul><ul><li>me on the web: </li></ul><ul><li>www.michaelneale.net, twitter.com/michaelneale, michaelneale.blogspot.com </li></ul>
  3. 3. Outline <ul><li>Define CEP? </li></ul><ul><li>Where is it used </li></ul><ul><li>Events and patterns </li></ul><ul><li>Sliding windows </li></ul><ul><li>Temporal reasoning </li></ul><ul><li>Defining an event “object” </li></ul><ul><li>Libraries: Drools, Esper </li></ul><ul><li>Sample scenario </li></ul>
  4. 4. Definition <ul><li>Treat “inputs” as “events” (time based) </li></ul><ul><li>Look for patterns/correlations to extract some meaning (“an inferred event/fact”) </li></ul><ul><li>Act on these patterns (return a result, perform a calculation, send a message etc) </li></ul><ul><li>Sometimes also called: </li></ul><ul><ul><li>Event Stream Processing </li></ul></ul><ul><ul><li>Temporal correlation </li></ul></ul>
  5. 5. eg <ul><li>Event: [Hand In Air] </li></ul><ul><li>Pattern: Lots of [Hand In Air] (possibly waving, as if they didn't care), happening in close (time and space) quarters... </li></ul><ul><li>Infer: A party </li></ul>
  6. 6. Huge number of events, but only a few of real interest Usually events are immutable Usually queries/rules have to run in reactive mode Strong temporal relationships between events Individual events are usually not important The composition and aggregation of events is important
  7. 7. Where is it used? <ul><li>Logistics: Package en-route tracking/warning </li></ul><ul><li>Stock market events/trades </li></ul><ul><ul><li>Provide real time alerts to end users </li></ul></ul><ul><li>Any where you have “needle in a haystack” problem, involving time based events </li></ul>
  8. 8. Events and patterns <ul><li>Events immutable** facts (objects/records) </li></ul><ul><ul><li>Eg a reading from a sensor, a stock value, a “fact-oid” </li></ul></ul><ul><li>Patterns are logic constraints (predicate/higher order statements) </li></ul><ul><ul><li>Think like the “where” part of a SQL query </li></ul></ul><ul><li>Patterns search for particular events... </li></ul>
  9. 9. Event instance of StockTick
  10. 10. Patterns <ul><li>Are used to detect specific events (using constraints and higher order logic) </li></ul><ul><li>Are used to detect co-incidences (over time periods) </li></ul><ul><li>Allow you to be declarative (once again, think SQL, but involving time) </li></ul>
  11. 11. Sliding time windows <ul><li>Declare what “window” of time you care about </li></ul><ul><li>Needed to narrow down from huge number of events </li></ul><ul><li>Help identify “co-incidences” which might be more then just co-incidences ! </li></ul><ul><li>Optional: maybe you care about ALL events, maybe just close together </li></ul>
  12. 12. Temporal reasoning <ul><li>A generalisation of “looking for co-incidences” of interest </li></ul><ul><li>Eg Hangover often follows from XXXX consumption, within certain time windows. Perhaps all of the time? (“forall”) Some of the time? (“exists”) </li></ul><ul><li>Look for sets of events, related in time, to infer what is really “going on” </li></ul><ul><li>Sets CEP apart from offline/batch processing </li></ul>
  13. 13. <ul><li>Event and Time semantics: </li></ul><ul><ul><li>Point in Time </li></ul></ul><ul><ul><li>Interval-based </li></ul></ul><ul><li>Unified semantics for event correlation over time </li></ul><ul><li>Temporal Constraints: </li></ul><ul><ul><li>Set of operators to express temporal relationship between events </li></ul></ul>Event semantics
  14. 14. Event semantics <ul><li>JF Allen defined a set of Time Event semantics: </li></ul>
  15. 15. Temporal operators <ul><li>After </li></ul><ul><li>Before </li></ul><ul><li>Coincides </li></ul><ul><li>During </li></ul><ul><li>Finishes </li></ul><ul><li>Finished By </li></ul><ul><li>Includes </li></ul><ul><li>Meets </li></ul><ul><li>Met By </li></ul><ul><li>Overlaps </li></ul><ul><li>Overlapped By </li></ul><ul><li>Starts </li></ul><ul><li>Started By </li></ul>
  16. 16. Defining events <ul><li>Generally classes defined in host language (eg java, C#) </li></ul><ul><li>Some extra metadata needed to define expiry, duration, timestamp etc (all optional). </li></ul><ul><li>Expiry important for memory management </li></ul><ul><ul><li>Large amounts of data </li></ul></ul><ul><ul><li>Why keep it in memory if not needed </li></ul></ul><ul><ul><li>Very stateful </li></ul></ul>
  17. 17. Defining events // declaring existing class import some.package.VoiceCall declare VoiceCall @role ( event ) @timestamp ( calltime ) @duration ( duration ) end // generating an event class declare StockTick @role ( event ) symbol : String price : double end
  18. 18. Drools Fusion <ul><li>jboss.org/drools </li></ul><ul><li>Submodule of Drools project </li></ul><ul><li>Uses rule language to express CEP patterns </li></ul><ul><ul><li>Same “drl” language as normal rules </li></ul></ul><ul><ul><li>Rules can take action directly, or be queries to return results </li></ul></ul>
  19. 19. <ul><li>rule &quot;Sound the alarm in case temperature rises above threshold&quot; </li></ul><ul><li>when </li></ul><ul><li>TemperatureThreshold( $max : max ) </li></ul><ul><li>Number( doubleValue > $max ) from accumulate( SensorReading( $temp : temperature ) over window:time( 10m ), average( $temp ) ) </li></ul><ul><li>then </li></ul><ul><li>// sound the alarm </li></ul>
  20. 20. Esper <ul><li>esper.codehaus.org </li></ul><ul><li>Takes “temporal SQL” approach </li></ul><ul><li>Return results for matches, a la SQL database (but its realtime events) </li></ul><ul><li>SQL skill portability ? </li></ul>
  21. 21. <ul><li>select fraud.accountNumber as accntNum, fraud.warning as warn, withdraw.amount as amount, </li></ul><ul><li>MAX(fraud.timestamp, withdraw.timestamp) as timestamp, 'withdrawlFraud' as desc </li></ul><ul><li>from FraudWarningEvent.win:time(30 min) as fraud, </li></ul><ul><li>WithdrawalEvent.win:time(30 sec) as withdraw </li></ul><ul><li>where fraud.accountNumber = withdraw.accountNumber </li></ul>
  22. 22. Deployment <ul><li>Usually a dedicated server/service endpoint </li></ul><ul><ul><li>(but could be embedded in your app if JVM friendly) </li></ul></ul><ul><li>(eg wrapped around a REST servlet, or MQ) </li></ul><ul><li>Sits there, gobbling up messages as fast as it can </li></ul>
  23. 23. Concurrency... <ul><li>Need for “realtime” performance </li></ul><ul><li>Need to make use of threads/processes </li></ul><ul><li>Use of “rulebase partitioning” to separate less coupled data to allow this </li></ul><ul><li>So: User doesn't have to worry </li></ul>
  24. 24. FedEx example <ul><li>Presented at JavaOne, Adobe MAX etc.. </li></ul><ul><li>FedEx Custom Critical: </li></ul><ul><li>1: En-Route Tracking Situational Awareness </li></ul><ul><li>2: Capacity Allocation Management </li></ul><ul><li>http://developers.sun.com/learning/javaoneonline/sessions/2009/pdf/TS-4475.pdf </li></ul>
  25. 29. A flex front end
  26. 31. Thanks, Q&A
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×