This document provides an overview of complex event processing (CEP). It defines CEP as treating inputs as events to look for patterns and correlations in order to extract meaning and act on inferred events. CEP is used in logistics, stock markets, and anywhere with a need to find patterns in large amounts of time-based event data. It discusses events, patterns, time windows, temporal reasoning, event definitions, CEP libraries like Drools and Esper, and provides an example of a FedEx tracking application.

1. CEP – complex event processing OSDC, Brisbane, November 2009 Michael Neale JBoss/Red Hat R&D

2. Michael Neale R&D w. JBoss (specialise in drools, now “cloud research”) Open source history (user -> fulltime developer “acquired” by jboss 2005). Thanks OSDC organisers ! me on the web: www.michaelneale.net, twitter.com/michaelneale, michaelneale.blogspot.com

3. Outline Define CEP? Where is it used Events and patterns Sliding windows Temporal reasoning Defining an event “object” Libraries: Drools, Esper Sample scenario

4. Definition Treat “inputs” as “events” (time based) Look for patterns/correlations to extract some meaning (“an inferred event/fact”) Act on these patterns (return a result, perform a calculation, send a message etc) Sometimes also called: Event Stream Processing Temporal correlation

5. eg Event: [Hand In Air] Pattern: Lots of [Hand In Air] (possibly waving, as if they didn't care), happening in close (time and space) quarters... Infer: A party

6. Huge number of events, but only a few of real interest Usually events are immutable Usually queries/rules have to run in reactive mode Strong temporal relationships between events Individual events are usually not important The composition and aggregation of events is important

7. Where is it used? Logistics: Package en-route tracking/warning Stock market events/trades Provide real time alerts to end users Any where you have “needle in a haystack” problem, involving time based events

8. Events and patterns Events immutable** facts (objects/records) Eg a reading from a sensor, a stock value, a “fact-oid” Patterns are logic constraints (predicate/higher order statements) Think like the “where” part of a SQL query Patterns search for particular events...

9. Event instance of StockTick

10. Patterns Are used to detect specific events (using constraints and higher order logic) Are used to detect co-incidences (over time periods) Allow you to be declarative (once again, think SQL, but involving time)

11. Sliding time windows Declare what “window” of time you care about Needed to narrow down from huge number of events Help identify “co-incidences” which might be more then just co-incidences ! Optional: maybe you care about ALL events, maybe just close together

12. Temporal reasoning A generalisation of “looking for co-incidences” of interest Eg Hangover often follows from XXXX consumption, within certain time windows. Perhaps all of the time? (“forall”) Some of the time? (“exists”) Look for sets of events, related in time, to infer what is really “going on” Sets CEP apart from offline/batch processing

13. Event and Time semantics: Point in Time Interval-based Unified semantics for event correlation over time Temporal Constraints: Set of operators to express temporal relationship between events Event semantics

14. Event semantics JF Allen defined a set of Time Event semantics:

15. Temporal operators After Before Coincides During Finishes Finished By Includes Meets Met By Overlaps Overlapped By Starts Started By

16. Defining events Generally classes defined in host language (eg java, C#) Some extra metadata needed to define expiry, duration, timestamp etc (all optional). Expiry important for memory management Large amounts of data Why keep it in memory if not needed Very stateful

17. Defining events // declaring existing class import some.package.VoiceCall declare VoiceCall @role ( event ) @timestamp ( calltime ) @duration ( duration ) end // generating an event class declare StockTick @role ( event ) symbol : String price : double end

18. Drools Fusion jboss.org/drools Submodule of Drools project Uses rule language to express CEP patterns Same “drl” language as normal rules Rules can take action directly, or be queries to return results

19. rule "Sound the alarm in case temperature rises above threshold" when TemperatureThreshold( $max : max ) Number( doubleValue > $max ) from accumulate( SensorReading( $temp : temperature ) over window:time( 10m ), average( $temp ) ) then // sound the alarm

20. Esper esper.codehaus.org Takes “temporal SQL” approach Return results for matches, a la SQL database (but its realtime events) SQL skill portability ?

21. select fraud.accountNumber as accntNum, fraud.warning as warn, withdraw.amount as amount, MAX(fraud.timestamp, withdraw.timestamp) as timestamp, 'withdrawlFraud' as desc from FraudWarningEvent.win:time(30 min) as fraud, WithdrawalEvent.win:time(30 sec) as withdraw where fraud.accountNumber = withdraw.accountNumber

22. Deployment Usually a dedicated server/service endpoint (but could be embedded in your app if JVM friendly) (eg wrapped around a REST servlet, or MQ) Sits there, gobbling up messages as fast as it can

23. Concurrency... Need for “realtime” performance Need to make use of threads/processes Use of “rulebase partitioning” to separate less coupled data to allow this So: User doesn't have to worry

24. FedEx example Presented at JavaOne, Adobe MAX etc.. FedEx Custom Critical: 1: En-Route Tracking Situational Awareness 2: Capacity Allocation Management http://developers.sun.com/learning/javaoneonline/sessions/2009/pdf/TS-4475.pdf

29. A flex front end

31. Thanks, Q&A