Complex Event Processing: What?, Why?, How?

3,271 views

Published on

Understand what Complex Event Processing (CEP) is
Where does CEP fit in SOA/EDA/BAM/BI/RDBMS
Learn about CEP use cases
Discover CEP adoption paths

Published in: Technology
0 Comments
7 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,271
On SlideShare
0
From Embeds
0
Number of Embeds
58
Actions
Shares
0
Downloads
0
Comments
0
Likes
7
Embeds 0
No embeds

No notes for slide

Complex Event Processing: What?, Why?, How?

  1. 1. Complex Event Processing Quoi, Pourquoi, Comment? Fabien Coppens Alexandre Vasseur ©OCTO Technology – Université d’été du Système d’information
  2. 2. Objectives • Understand what Complex Event Processing (CEP) is • Where does CEP fit in SOA/EDA/BAM/BI/RDBMS • Learn about CEP use cases • Discover CEP adoption paths
  3. 3. Speaker Qualifications • Fabien Coppens • Senior Architect • CEP user • Alexandre Vasseur • Middleware technology evangelist • Esper co-lead & early CEP fan
  4. 4. The « Right-Time » Enterprise Situation • Business is data-driven – More data, more sources Events – More complex situational scenarios • Right-time is Time – A business imperative – A technology challenge Push • Strong coupling! Loosely coupled • Request/Response! Time & • Data stored and later analyzed Event-Driven • Moving requirements Action/Decision
  5. 5. SOA + EDA + XTP + Edge + BAM/BI Convergence: Event Processing « By 2011, a new generation of application platforms stemming from the convergence of extreme transaction processing (XTP) technologies will supersede Java EE and .NET as the platform of choice for large-scale, business-critical operational applications.“ Gartner, July 2007 Large scale SOAs RFID BAM & monitoring Financial Services Sensors Fraud detection Location/presence based services Security Right-Time + Events
  6. 6. Event Processing Simple, Streaming, Complex / Business EP • Simple • Consumer/Producer, JMS, ESBs, EIP – Route, Filter, Enrich – Stateless & Timeless • Streaming • How? – min,max,avg – « Event Stream Processing » – Sliding data windows – Statefu-less / Timefu-less • Complex / Business • How?? – Many streams – Continuous joins and indexing – Causality – Time as first class citizen A followed by B • Temporal requests within t minutes – Statefull / Timefull – Different rates – Historical data and real time data When [CEP] then […]
  7. 7. CEP’s Friends • Edge – Produce zillions of events • SOA – Interconnect, mediate, leverage and reuse • XTP – Scale throughput and data volumes as required – Right-time down to sub-millisecond where needed • Rules engine When [CEP] then […] If … then […] CEP Situational & Actionable Awareness • BAM & BI (or back to SOA) – Operational applications
  8. 8. CEP ~ Database 2.0 ? • RDBMS • CEP – Store data (a lot) – Store rules – Handle queries – Handle data – SQL (ie abstracted) – EPL: Event Processing Language • Request/Response • Subscribe/Notify • Concept of Time • Time & causality • Right time • « Continuous query » CEP Server Database EPL Queries SQL Queries CEP
  9. 9. Event Processing Language & CEP Server • EPL: For right-time events, what SQL is to historical data – Best practices, rapid prototyping, tooling, less custom code • CEP server: Middleware to deal with recurrent complexity – indexing, state management, scalability, management, connectivity etc Transports Is EPL expressive enough Dashboards (build vs buy) In & Out Is EPL a standard CEP Application (vs SQL) Source + CEP as EPL + Sink Is it a specialized RDBMS or a true engine? Does it require a Runtime Management Runtime Management Record / Replay / Test Record / Replay / Test Time & RDBMS? Filter, Aggregate Causality Manageable? Sliding windows Scalable Continuous execution Interoperate? State management, threading Interrop. High Availability
  10. 10. #1: Algorithmic trading • Volume Weighted Average select ticker, avg(price), count(*), sum(price) from MarketData(ticker=‘GOOG').win:length(1000) Simple Event Processing Stream Processing (Timefull) • Example CEP server performance figures – 1 CEP server, Java based, TCP in/out – 1000 CEP statements – 500 000 evt/s [100 Mbit/s] – Latency: avg 2.5µs, 99% < 10µs – 4 core 2GHz, 65% load avg
  11. 11. #1: Build vs Buy? Avg price / last 2 ticks, per ticker MarketData(GOOG, 50) list = new MyList(GOOG,2) /*2 = size*/ // on event MarketData(MSFT, 25) list.add(tick) // some tick discarded // as list size was MarketData(GOOG, 30) // harcoded with size 2 //TODO: do not hardcode GOOG:40 // recompute avg MarketData(GOOG, 40) //TODO: optimize this send computeAvg(list) GOOG:35 Receive market data • Functional specification? • Abstraction for user defined rules? Output / trigger actions • Performance & monitoring etc?
  12. 12. #2: Fraud and compliance, risk management Context – FX trading – Event { Trade direction, amount, user id, account, currencies traded } – 200 000 evt/day, 50 rules, ~5% of triggering Two (or more) buy trades followed by ( ) a sale within the next hour between 2 parties select e1, e2, final from pattern [ every e1=FXTrade(userId in (100,101)) (e2=FXTrade(userId in (100,101), dir = e1.dir) where timer:within(8 hours) ) final=FXTrade(userId in (100,101), dir != e1.dir) where timer:within(1 hour) ]
  13. 13. #3: Location based services Context: • Mobile asset management (Command & Control systems) • Tracking delay between real time position and planned position – 20% time left to reach destination, 10% delta from planned position • Continuous join with reference (passive) data select id, lat, lon, time-ref.time, Utils.distance(lat,lon,ref.lat,ref.lon) as distance from GPSPosition, sql:RefDB [ select lat, lon, time from Plan where id={id} and stage=‘JFK’ ] as ref where time<0.8*ref.time and Utils.distance(lat,lon,ref.lat,ref.lon)>1.1*SPEED*(0.8*ref.time-time)
  14. 14. #4: Trading as SaaS : Strateer • Algo trading as SaaS • CEP is a core enabler • Users can configure their own trading strategies • ~20 CEP rules templates provided – Each configurable by end users • Power users can create their own rules • Esper CEP + JMS + MarketData connectors + RIA
  15. 15. #4: Strateer use case 1. Drag & Drop 2. your own automated Test it the way strategy fund managers 3. do 5. Then run it 4. Share and on Strateer: compare Get trading 1000s of data results with alerts to you sources, real- community anytime, time engine. anywhere. Strateer is algorithmic trading for individual investors
  16. 16. #4: Strateer architecture
  17. 17. Solutions (excerpt) Single use case platform (trading, Event Processing = Middleware compliance, BI ..) xxx Event Processing x = Simple / Stream / Complex / Rules ? Mandates a DB? In-memory vs hidden DB vs DB Latency / Throughput 1 to 1000 if not more range Of the shelf? Interoperate? Open platform? Proven / Innovating … Maturity … Cost (Buy + Learn + Operate) …
  18. 18. Adoption: Risks & Benefits “CEP is mature? CEP is really not ESP? CEP is really event-driven SOA? CEP is really real-time BI? CEP is really low latency, high throughput, white-box COTs algo trading? CEP is really not a type of BPM? CEP is not really for detecting complex events? Complex does not really mean complex?” [as seen on blogs] Use case driven ? Build vs Buy SOA/BI/BAM related Orthogonal processing (discover & monitor) vs path critical processing (act upon)
  19. 19. Links • http://esper.codehaus.org • http://www.espertech.com • http://complexevents.com • And many other googling around or asking your favorite vendors (!)

×