The document discusses Complex Event Processing (CEP), explaining that CEP is an emerging technology that allows processing of streaming event data in real-time to detect complex patterns and situations. Several use cases for CEP are presented, including algorithmic trading, fraud detection, and location-based services. The document also covers CEP adoption paths and debates around CEP, such as whether it is mature enough and how it relates to other technologies.

1. Complex Event Processing Quoi, Pourquoi, Comment? Fabien Coppens Alexandre Vasseur ©OCTO Technology – Université d’été du Système d’information

2. Objectives Understand what Complex Event Processing (CEP) is Where does CEP fit in SOA/EDA/BAM/BI/RDBMS Learn about CEP use cases Discover CEP adoption paths

3. Speaker Qualifications Fabien Coppens Senior Architect CEP user Alexandre Vasseur Middleware technology evangelist Esper co-lead & early CEP fan

4. The « Right-Time » Enterprise Business is data-driven More data, more sources More complex situational scenarios Right-time is A business imperative A technology challenge Strong coupling! Request/Response! Data stored and later analyzed Moving requirements Situation Action/Decision Events Time Push Loosely coupled Time & Event-Driven

5. « By 2011, a new generation of application platforms stemming from the convergence of extreme transaction processing (XTP) technologies will supersede Java EE and .NET as the platform of choice for large-scale, business-critical operational applications .“ Gartner, July 2007 SOA + EDA + XTP + Edge + BAM/BI Convergence: Event Processing Financial Services Fraud detection RFID BAM & monitoring Sensors Location/presence based services Security Large scale SOAs Right-Time + Events

6. Event Processing Simple, Streaming, Complex / Business EP Simple Route, Filter, Enrich Streaming min,max,avg Sliding data windows Complex / Business Many streams Causality A followed by B within t minutes Different rates Historical data and real time data Consumer/Producer, JMS, ESBs, EIP Stateless & Timeless How? « Event Stream Processing » Statefu-less / Timefu-less How?? Continuous joins and indexing Time as first class citizen Temporal requests Statefull / Timefull When [CEP] then […]

7. CEP’s Friends Edge Produce zillions of events SOA Interconnect, mediate, leverage and reuse XTP Scale throughput and data volumes as required Right-time down to sub-millisecond where needed BAM & BI (or back to SOA) Operational applications CEP Situational & Actionable Awareness Rules engine When [CEP] then […] If … then […]

8. CEP ~ Database 2.0 ? RDBMS Store data (a lot) Handle queries SQL (ie abstracted) Request/Response Concept of Time Right time CEP Store rules Handle data EPL: Event Processing Language Subscribe/Notify Time & causality « Continuous query » SQL Queries CEP Server EPL Queries CEP Database

9. Event Processing Language & CEP Server EPL: For right-time events, what SQL is to historical data Best practices, rapid prototyping, tooling, less custom code CEP server: Middleware to deal with recurrent complexity indexing, state management, scalability, management, connectivity etc ? CEP Application Source + CEP as EPL + Sink Is EPL expressive enough (build vs buy) Is EPL a standard (vs SQL) Is it a specialized RDBMS or a true engine? Does it require a RDBMS? Manageable? Scalable Interoperate? Time & Causality Filter, Aggregate Sliding windows Continuous execution State management, threading Transports In & Out Interrop. Dashboards High Availability Runtime Management Record / Replay / Test

10. #1: Algorithmic trading Volume Weighted Average select ticker, avg (price), count (*), sum (price) from MarketData(ticker=‘GOOG'). win:length (1000) Example CEP server performance figures 1 CEP server, Java based, TCP in/out 1000 CEP statements 500 000 evt/s [100 Mbit/s] Latency: avg 2.5 µ s, 99% < 10 µ s 4 core 2GHz, 65% load avg Simple Event Processing Stream Processing (Timefull)

11. #1: Build vs Buy? MarketData(GOOG, 50) MarketData(MSFT, 25) MarketData(GOOG, 30) MarketData(GOOG, 40) Avg price / last 2 ticks, per ticker GOOG:40 GOOG:35 Receive market data Output / trigger actions Functional specification? Abstraction for user defined rules? Performance & monitoring etc? list = new MyList(GOOG,2) /*2 = size*/ // on event list.add(tick) // some tick discarded // as list size was // harcoded with size 2 //TODO: do not hardcode // recompute avg //TODO: optimize this send computeAvg(list)

12. #2: Fraud and compliance, risk management Context FX trading Event { Trade direction, amount, user id, account, currencies traded } 200 000 evt/day, 50 rules, ~5% of triggering Two (or more) buy trades followed by (  ) a sale within the next hour between 2 parties select e1, e2, final from pattern [ every e1=FXTrade(userId in (100,101))  (e2=FXTrade(userId in (100,101), dir = e1.dir) where timer:within (8 hours ) )  final=FXTrade(userId in (100,101), dir != e1.dir) where timer:within (1 hour ) ]

13. #3: Location based services Context: Mobile asset management (Command & Control systems) Tracking delay between real time position and planned position 20% time left to reach destination, 10% delta from planned position Continuous join with reference (passive) data select id, lat, lon, time-ref.time, Utils.distance (lat,lon,ref.lat,ref.lon) as distance from GPSPosition, sql:RefDB [ select lat, lon, time from Plan where id={id} and stage=‘JFK’ ] as ref where time<0.8*ref.time and Utils.distance(lat,lon,ref.lat,ref.lon)>1.1*SPEED*(0.8*ref.time-time)

14. #4: Trading as SaaS : Strateer Algo trading as SaaS CEP is a core enabler Users can configure their own trading strategies ~20 CEP rules templates provided Each configurable by end users Power users can create their own rules Esper CEP + JMS + MarketData connectors + RIA

15. #4: Strateer use case Strateer is algorithmic trading for individual investors 1. Drag & Drop your own automated strategy 2. Test it the way fund managers do 3. Then run it on Strateer: 1000s of data sources, real-time engine. 4. Get trading alerts to you anytime, anywhere. 5. Share and compare results with community

16. #4: Strateer architecture

17. Solutions (excerpt) Single use case platform (trading, compliance, BI ..) xxx Event Processing Mandates a DB? Latency / Throughput Of the shelf? Proven / Innovating Maturity Cost (Buy + Learn + Operate) Event Processing = Middleware x = Simple / Stream / Complex / Rules In-memory vs hidden DB vs DB 1 to 1000 if not more range Interoperate? Open platform? … … … ?

18. Adoption: Risks & Benefits ? “ CEP is mature? CEP is really not ESP? CEP is really event-driven SOA? CEP is really real-time BI? CEP is really low latency, high throughput, white-box COTs algo trading? CEP is really not a type of BPM? CEP is not really for detecting complex events? Complex does not really mean complex?” [as seen on blogs] Use case driven Build vs Buy SOA/BI/BAM related Orthogonal processing (discover & monitor) vs path critical processing (act upon)

19. Links http://esper.codehaus.org http://www.espertech.com http://complexevents.com And many other googling around or asking your favorite vendors (!)