Complex Event Processing: What?, Why?, How?


Published on

July 2008, Introduction session to CEP from various angles, and short case studies around Esper

Published in: Technology, Economy & Finance

Complex Event Processing: What?, Why?, How?

  1. 1. Complex Event Processing Quoi, Pourquoi, Comment? Fabien Coppens Alexandre Vasseur ©OCTO Technology – Université d’été du Système d’information
  2. 2. Objectives <ul><li>Understand what Complex Event Processing (CEP) is </li></ul><ul><li>Where does CEP fit in SOA/EDA/BAM/BI/RDBMS </li></ul><ul><li>Learn about CEP use cases </li></ul><ul><li>Discover CEP adoption paths </li></ul>
  3. 3. Speaker Qualifications <ul><li>Fabien Coppens </li></ul><ul><li>Senior Architect </li></ul><ul><li>CEP user </li></ul><ul><li>Alexandre Vasseur </li></ul><ul><li>Middleware technology evangelist </li></ul><ul><li>Esper co-lead & early CEP fan </li></ul>
  4. 4. The « Right-Time » Enterprise <ul><li>Business is data-driven </li></ul><ul><ul><li>More data, more sources </li></ul></ul><ul><ul><li>More complex situational scenarios </li></ul></ul><ul><li>Right-time is </li></ul><ul><ul><li>A business imperative </li></ul></ul><ul><ul><li>A technology challenge </li></ul></ul><ul><ul><ul><li>Strong coupling! </li></ul></ul></ul><ul><ul><ul><li>Request/Response! </li></ul></ul></ul><ul><ul><ul><li>Data stored and later analyzed </li></ul></ul></ul><ul><ul><ul><li>Moving requirements </li></ul></ul></ul>Situation Action/Decision Events Time Push Loosely coupled Time & Event-Driven
  5. 5. <ul><li>« By 2011, a new generation of application platforms stemming from the convergence of extreme transaction processing (XTP) technologies will supersede Java EE and .NET as the platform of choice for large-scale, business-critical operational applications .“ Gartner, July 2007 </li></ul>SOA + EDA + XTP + Edge + BAM/BI Convergence: Event Processing Financial Services Fraud detection RFID BAM & monitoring Sensors Location/presence based services Security Large scale SOAs Right-Time + Events
  6. 6. Event Processing Simple, Streaming, Complex / Business EP <ul><li>Simple </li></ul><ul><ul><li>Route, Filter, Enrich </li></ul></ul><ul><li>Streaming </li></ul><ul><ul><li>min,max,avg </li></ul></ul><ul><ul><li>Sliding data windows </li></ul></ul><ul><li>Complex / Business </li></ul><ul><ul><li>Many streams </li></ul></ul><ul><ul><li>Causality A followed by B within t minutes </li></ul></ul><ul><ul><li>Different rates </li></ul></ul><ul><ul><li>Historical data and real time data </li></ul></ul><ul><li>Consumer/Producer, JMS, ESBs, EIP </li></ul><ul><ul><li>Stateless & Timeless </li></ul></ul><ul><li>How? </li></ul><ul><ul><li>« Event Stream Processing » </li></ul></ul><ul><ul><li>Statefu-less / Timefu-less </li></ul></ul><ul><li>How?? </li></ul><ul><ul><li>Continuous joins and indexing </li></ul></ul><ul><ul><li>Time as first class citizen </li></ul></ul><ul><ul><ul><li>Temporal requests </li></ul></ul></ul><ul><ul><li>Statefull / Timefull </li></ul></ul>When [CEP] then […]
  7. 7. CEP’s Friends <ul><li>Edge </li></ul><ul><ul><li>Produce zillions of events </li></ul></ul><ul><li>SOA </li></ul><ul><ul><li>Interconnect, mediate, leverage and reuse </li></ul></ul><ul><li>XTP </li></ul><ul><ul><li>Scale throughput and data volumes as required </li></ul></ul><ul><ul><li>Right-time down to sub-millisecond where needed </li></ul></ul><ul><li>BAM & BI (or back to SOA) </li></ul><ul><ul><li>Operational applications </li></ul></ul>CEP Situational & Actionable Awareness <ul><li>Rules engine </li></ul>When [CEP] then […] If … then […]
  8. 8. CEP ~ Database 2.0 ? <ul><li>RDBMS </li></ul><ul><ul><li>Store data (a lot) </li></ul></ul><ul><ul><li>Handle queries </li></ul></ul><ul><ul><li>SQL (ie abstracted) </li></ul></ul><ul><li>Request/Response </li></ul><ul><li>Concept of Time </li></ul><ul><li>Right time </li></ul><ul><li>CEP </li></ul><ul><ul><li>Store rules </li></ul></ul><ul><ul><li>Handle data </li></ul></ul><ul><ul><li>EPL: Event Processing Language </li></ul></ul><ul><li>Subscribe/Notify </li></ul><ul><li>Time & causality </li></ul><ul><li>« Continuous query » </li></ul>SQL Queries CEP Server EPL Queries CEP Database
  9. 9. Event Processing Language & CEP Server <ul><li>EPL: For right-time events, what SQL is to historical data </li></ul><ul><ul><li>Best practices, rapid prototyping, tooling, less custom code </li></ul></ul><ul><li>CEP server: Middleware to deal with recurrent complexity </li></ul><ul><ul><li>indexing, state management, scalability, management, connectivity etc </li></ul></ul>? CEP Application Source + CEP as EPL + Sink Is EPL expressive enough (build vs buy) Is EPL a standard (vs SQL) Is it a specialized RDBMS or a true engine? Does it require a RDBMS? Manageable? Scalable Interoperate? Time & Causality Filter, Aggregate Sliding windows Continuous execution State management, threading Transports In & Out Interrop. Dashboards High Availability Runtime Management Record / Replay / Test
  10. 10. #1: Algorithmic trading <ul><li>Volume Weighted Average </li></ul><ul><li>select ticker, avg (price), count (*), sum (price) </li></ul><ul><li>from MarketData(ticker=‘GOOG'). win:length (1000) </li></ul><ul><li>Example CEP server performance figures </li></ul><ul><ul><li>1 CEP server, Java based, TCP in/out </li></ul></ul><ul><ul><li>1000 CEP statements </li></ul></ul><ul><ul><li>500 000 evt/s [100 Mbit/s] </li></ul></ul><ul><ul><li>Latency: avg 2.5 µ s, 99% < 10 µ s </li></ul></ul><ul><ul><li>4 core 2GHz, 65% load avg </li></ul></ul>Simple Event Processing Stream Processing (Timefull)
  11. 11. #1: Build vs Buy? MarketData(GOOG, 50) MarketData(MSFT, 25) MarketData(GOOG, 30) MarketData(GOOG, 40) Avg price / last 2 ticks, per ticker GOOG:40 GOOG:35 Receive market data Output / trigger actions <ul><li>Functional specification? </li></ul><ul><li>Abstraction for user defined rules? </li></ul><ul><li>Performance & monitoring etc? </li></ul>list = new MyList(GOOG,2) /*2 = size*/ // on event list.add(tick) // some tick discarded // as list size was // harcoded with size 2 //TODO: do not hardcode // recompute avg //TODO: optimize this send computeAvg(list)
  12. 12. #2: Fraud and compliance, risk management <ul><li>Context </li></ul><ul><ul><li>FX trading </li></ul></ul><ul><ul><li>Event { Trade direction, amount, user id, account, currencies traded } </li></ul></ul><ul><ul><li>200 000 evt/day, 50 rules, ~5% of triggering </li></ul></ul><ul><li>Two (or more) buy trades followed by (  ) a sale within the next hour between 2 parties </li></ul><ul><li>select e1, e2, final from pattern [ </li></ul><ul><li>every e1=FXTrade(userId in (100,101)) </li></ul><ul><li> (e2=FXTrade(userId in (100,101), dir = e1.dir) </li></ul><ul><li>where timer:within (8 hours ) </li></ul><ul><li>)  final=FXTrade(userId in (100,101), dir != e1.dir) </li></ul><ul><li>where timer:within (1 hour ) ] </li></ul>
  13. 13. #3: Location based services <ul><li>Context: </li></ul><ul><li>Mobile asset management (Command & Control systems) </li></ul><ul><li>Tracking delay between real time position and planned position </li></ul><ul><ul><li>20% time left to reach destination, 10% delta from planned position </li></ul></ul><ul><li>Continuous join with reference (passive) data </li></ul><ul><li>select id, lat, lon, time-ref.time, </li></ul><ul><li>Utils.distance (lat,lon,,ref.lon) </li></ul><ul><li>as distance </li></ul><ul><li>from GPSPosition, </li></ul><ul><li>sql:RefDB [ </li></ul><ul><li>select lat, lon, time from Plan </li></ul><ul><li>where id={id} and stage=‘JFK’ </li></ul><ul><li>] as ref </li></ul><ul><li>where time<0.8*ref.time and </li></ul><ul><li>Utils.distance(lat,lon,,ref.lon)>1.1*SPEED*(0.8*ref.time-time) </li></ul>
  14. 14. #4: Trading as SaaS : Strateer <ul><li>Algo trading as SaaS </li></ul><ul><li>CEP is a core enabler </li></ul><ul><li>Users can configure their own trading strategies </li></ul><ul><li>~20 CEP rules templates provided </li></ul><ul><ul><li>Each configurable by end users </li></ul></ul><ul><li>Power users can create their own rules </li></ul><ul><li>Esper CEP + JMS + MarketData connectors + RIA </li></ul>
  15. 15. #4: Strateer use case Strateer is algorithmic trading for individual investors 1. Drag & Drop your own automated strategy 2. Test it the way fund managers do 3. Then run it on Strateer: 1000s of data sources, real-time engine. 4. Get trading alerts to you anytime, anywhere. 5. Share and compare results with community
  16. 16. #4: Strateer architecture
  17. 17. Solutions (excerpt) Single use case platform (trading, compliance, BI ..) xxx Event Processing Mandates a DB? Latency / Throughput Of the shelf? Proven / Innovating Maturity Cost (Buy + Learn + Operate) Event Processing = Middleware x = Simple / Stream / Complex / Rules In-memory vs hidden DB vs DB 1 to 1000 if not more range Interoperate? Open platform? … … … ?
  18. 18. Adoption: Risks & Benefits ? “ CEP is mature? CEP is really not ESP? CEP is really event-driven SOA? CEP is really real-time BI? CEP is really low latency, high throughput, white-box COTs algo trading? CEP is really not a type of BPM? CEP is not really for detecting complex events? Complex does not really mean complex?” [as seen on blogs] Use case driven Build vs Buy SOA/BI/BAM related Orthogonal processing (discover & monitor) vs path critical processing (act upon)
  19. 19. Links <ul><li> </li></ul><ul><li> </li></ul><ul><li> </li></ul><ul><li>And many other googling around or asking your favorite vendors (!) </li></ul>