• Save
Complex Event Processing: What?, Why?, How?
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Complex Event Processing: What?, Why?, How?

on

  • 5,397 views

July 2008, Introduction session to CEP from various angles, and short case studies around Esper

July 2008, Introduction session to CEP from various angles, and short case studies around Esper

Statistics

Views

Total Views
5,397
Views on SlideShare
5,359
Embed Views
38

Actions

Likes
10
Downloads
0
Comments
0

2 Embeds 38

http://jisi.dreamblog.jp 24
http://www.slideshare.net 14

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Complex Event Processing: What?, Why?, How? Presentation Transcript

  • 1. Complex Event Processing Quoi, Pourquoi, Comment? Fabien Coppens Alexandre Vasseur ©OCTO Technology – Université d’été du Système d’information
  • 2. Objectives
    • Understand what Complex Event Processing (CEP) is
    • Where does CEP fit in SOA/EDA/BAM/BI/RDBMS
    • Learn about CEP use cases
    • Discover CEP adoption paths
  • 3. Speaker Qualifications
    • Fabien Coppens
    • Senior Architect
    • CEP user
    • Alexandre Vasseur
    • Middleware technology evangelist
    • Esper co-lead & early CEP fan
  • 4. The « Right-Time » Enterprise
    • Business is data-driven
      • More data, more sources
      • More complex situational scenarios
    • Right-time is
      • A business imperative
      • A technology challenge
        • Strong coupling!
        • Request/Response!
        • Data stored and later analyzed
        • Moving requirements
    Situation Action/Decision Events Time Push Loosely coupled Time & Event-Driven
  • 5.
    • « By 2011, a new generation of application platforms stemming from the convergence of extreme transaction processing (XTP) technologies will supersede Java EE and .NET as the platform of choice for large-scale, business-critical operational applications .“ Gartner, July 2007
    SOA + EDA + XTP + Edge + BAM/BI Convergence: Event Processing Financial Services Fraud detection RFID BAM & monitoring Sensors Location/presence based services Security Large scale SOAs Right-Time + Events
  • 6. Event Processing Simple, Streaming, Complex / Business EP
    • Simple
      • Route, Filter, Enrich
    • Streaming
      • min,max,avg
      • Sliding data windows
    • Complex / Business
      • Many streams
      • Causality A followed by B within t minutes
      • Different rates
      • Historical data and real time data
    • Consumer/Producer, JMS, ESBs, EIP
      • Stateless & Timeless
    • How?
      • « Event Stream Processing »
      • Statefu-less / Timefu-less
    • How??
      • Continuous joins and indexing
      • Time as first class citizen
        • Temporal requests
      • Statefull / Timefull
    When [CEP] then […]
  • 7. CEP’s Friends
    • Edge
      • Produce zillions of events
    • SOA
      • Interconnect, mediate, leverage and reuse
    • XTP
      • Scale throughput and data volumes as required
      • Right-time down to sub-millisecond where needed
    • BAM & BI (or back to SOA)
      • Operational applications
    CEP Situational & Actionable Awareness
    • Rules engine
    When [CEP] then […] If … then […]
  • 8. CEP ~ Database 2.0 ?
    • RDBMS
      • Store data (a lot)
      • Handle queries
      • SQL (ie abstracted)
    • Request/Response
    • Concept of Time
    • Right time
    • CEP
      • Store rules
      • Handle data
      • EPL: Event Processing Language
    • Subscribe/Notify
    • Time & causality
    • « Continuous query »
    SQL Queries CEP Server EPL Queries CEP Database
  • 9. Event Processing Language & CEP Server
    • EPL: For right-time events, what SQL is to historical data
      • Best practices, rapid prototyping, tooling, less custom code
    • CEP server: Middleware to deal with recurrent complexity
      • indexing, state management, scalability, management, connectivity etc
    ? CEP Application Source + CEP as EPL + Sink Is EPL expressive enough (build vs buy) Is EPL a standard (vs SQL) Is it a specialized RDBMS or a true engine? Does it require a RDBMS? Manageable? Scalable Interoperate? Time & Causality Filter, Aggregate Sliding windows Continuous execution State management, threading Transports In & Out Interrop. Dashboards High Availability Runtime Management Record / Replay / Test
  • 10. #1: Algorithmic trading
    • Volume Weighted Average
    • select ticker, avg (price), count (*), sum (price)
    • from MarketData(ticker=‘GOOG'). win:length (1000)
    • Example CEP server performance figures
      • 1 CEP server, Java based, TCP in/out
      • 1000 CEP statements
      • 500 000 evt/s [100 Mbit/s]
      • Latency: avg 2.5 µ s, 99% < 10 µ s
      • 4 core 2GHz, 65% load avg
    Simple Event Processing Stream Processing (Timefull)
  • 11. #1: Build vs Buy? MarketData(GOOG, 50) MarketData(MSFT, 25) MarketData(GOOG, 30) MarketData(GOOG, 40) Avg price / last 2 ticks, per ticker GOOG:40 GOOG:35 Receive market data Output / trigger actions
    • Functional specification?
    • Abstraction for user defined rules?
    • Performance & monitoring etc?
    list = new MyList(GOOG,2) /*2 = size*/ // on event list.add(tick) // some tick discarded // as list size was // harcoded with size 2 //TODO: do not hardcode // recompute avg //TODO: optimize this send computeAvg(list)
  • 12. #2: Fraud and compliance, risk management
    • Context
      • FX trading
      • Event { Trade direction, amount, user id, account, currencies traded }
      • 200 000 evt/day, 50 rules, ~5% of triggering
    • Two (or more) buy trades followed by (  ) a sale within the next hour between 2 parties
    • select e1, e2, final from pattern [
    • every e1=FXTrade(userId in (100,101))
    •  (e2=FXTrade(userId in (100,101), dir = e1.dir)
    • where timer:within (8 hours )
    • )  final=FXTrade(userId in (100,101), dir != e1.dir)
    • where timer:within (1 hour ) ]
  • 13. #3: Location based services
    • Context:
    • Mobile asset management (Command & Control systems)
    • Tracking delay between real time position and planned position
      • 20% time left to reach destination, 10% delta from planned position
    • Continuous join with reference (passive) data
    • select id, lat, lon, time-ref.time,
    • Utils.distance (lat,lon,ref.lat,ref.lon)
    • as distance
    • from GPSPosition,
    • sql:RefDB [
    • select lat, lon, time from Plan
    • where id={id} and stage=‘JFK’
    • ] as ref
    • where time<0.8*ref.time and
    • Utils.distance(lat,lon,ref.lat,ref.lon)>1.1*SPEED*(0.8*ref.time-time)
  • 14. #4: Trading as SaaS : Strateer
    • Algo trading as SaaS
    • CEP is a core enabler
    • Users can configure their own trading strategies
    • ~20 CEP rules templates provided
      • Each configurable by end users
    • Power users can create their own rules
    • Esper CEP + JMS + MarketData connectors + RIA
  • 15. #4: Strateer use case Strateer is algorithmic trading for individual investors 1. Drag & Drop your own automated strategy 2. Test it the way fund managers do 3. Then run it on Strateer: 1000s of data sources, real-time engine. 4. Get trading alerts to you anytime, anywhere. 5. Share and compare results with community
  • 16. #4: Strateer architecture
  • 17. Solutions (excerpt) Single use case platform (trading, compliance, BI ..) xxx Event Processing Mandates a DB? Latency / Throughput Of the shelf? Proven / Innovating Maturity Cost (Buy + Learn + Operate) Event Processing = Middleware x = Simple / Stream / Complex / Rules In-memory vs hidden DB vs DB 1 to 1000 if not more range Interoperate? Open platform? … … … ?
  • 18. Adoption: Risks & Benefits ? “ CEP is mature? CEP is really not ESP? CEP is really event-driven SOA? CEP is really real-time BI? CEP is really low latency, high throughput, white-box COTs algo trading? CEP is really not a type of BPM? CEP is not really for detecting complex events? Complex does not really mean complex?” [as seen on blogs] Use case driven Build vs Buy SOA/BI/BAM related Orthogonal processing (discover & monitor) vs path critical processing (act upon)
  • 19. Links
    • http://esper.codehaus.org
    • http://www.espertech.com
    • http://complexevents.com
    • And many other googling around or asking your favorite vendors (!)