Your SlideShare is downloading. ×
Esper - CEP Engine
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Esper - CEP Engine

14,099
views

Published on

Tech Talk about Esper - Complex Event Processing Engine in Java. …

Tech Talk about Esper - Complex Event Processing Engine in Java.

Presented at JTeam Tech Meeting by Aparna Chaudhary

Published in: Technology

0 Comments
62 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
14,099
On Slideshare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
921
Comments
0
Likes
62
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Esper The Power to Predict JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 2. Agenda • Predictive Business • Complex Event Processing • Esper • Demo • Questions JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 3. Predictive Business "In the future, every company must be able to recognize what is coming early enough to take evasive action or steer towards an emerging opportunity."[Frederick W. Smith-CEO FedEx] JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 4. Business Demands High throughput Complexity High Availability Low latency Insight JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 5. Conventional Models • RDBMS • Adds delays into data storage • Uses Polling Model • Wasteful Use of Resources JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 6. Conventional Models • Messaging Systems • Process Unitary Events • No Correlation support JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 7. Agenda • Predictive Business • Complex Event Processing • Esper • Demo • Questions JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 8. Complex Event Processing JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 9. Concepts - Event An event is a piece of data that represents that something happened in the real world, or in software system. e.g. Stock Tick, Weather, UserClick JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 10. Concepts - Complex Event An event that could only happen when lots of other events happen. e.g. Housing Price Drop JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 11. Concepts - Event Stream A linearly ordered (often by time) sequence of events forms Event Stream. e.g. Click Stream JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 12. Concepts - Event Cloud A partially ordered set of events form Event Cloud e.g. Set of all stock trades for GOOG within a 5 minute time window is an Event Stream. While all Stocks sold in a business day is an Event Cloud. JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 13. Concepts - Event Cloud JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 14. ESP and CEP Event Stream Processing is focused more on high-speed querying of data in streams of events and applying mathematical algorithms to the event data. Complex Event Processing includes event data analysis, but places emphasis on patterns of events, and abstracting and simplifying information in the patterns. JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 15. CEP Engine An event processor is an application that performs operations on event objects, including creating, reading, transforming, aggregating, correlating or removing them. JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 16. CEP Engine JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 17. CEP Engine Characteristics 1. Keep the data moving 2. Query on streams 3. Handle Stream Imperfections 4. Integrate Stored and Streaming Data 5. Guarantee Data Safety and Availability JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 18. CEP Engine Characteristics 6. Partition and scale applications automatically 7. Process and Respond instantaneously 8. Interoperability JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 19. CEP in Application Architecture JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 20. Commercial CEP Products • Weblogic Event Server • Coral8 • TIBCO • IBM • StreamBase • Progress JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 21. CEP/EDA and SOA [1/3] EDA is SOA 2.0 EDA is Proactive SOA EDA and SOA Complementary JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 22. CEP/EDA and SOA [2/3] JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 23. CEP/EDA and SOA [3/3] JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 24. Real World Applications Capital Markets BAM - Business Activity Monitoring Supply Chain Management Flight Operations Monitoring Fraud Detection JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 25. Agenda • Predictive Business • Complex Event Processing • Esper • Demo • Questions JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 26. Esper • Dual Licensing model • Open Source GPL • Esper Tech • Q2/2007 BEA Partner JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 27. Good Documentation • Nice reference document • Patterns catalogue available online • Lots of examples JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 28. Lightweight • ANTLR • CGLIB • LOG4J JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 29. Ohloh Analysis Summary JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 30. Pluggability • Works with any Java Process • OSGI Enabled JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 31. Architecture ESPER Engine EPServiceProvider Event EPL JDB Objects Statements Confi C gurat Adap ion Listeners ter Subscribers JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 32. API Overview • EPServiceProvider • Acts as Engine • EPStatement • Queries written in EQL • UpdateListener JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 33. Event • POJOs • Key value pairs (java.util.Map) • XML Documents JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 34. Event JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 35. Event Properties Type Syntax Example Simple name sensor Indexed name[index] sensor[0] Mapped name(key) sensor(‘light’) Nested name.nested sensor.value Name JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 36. UpdateListener Receives updated data as soon as it is processed for the statement. JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 37. Subscriber • Must provide a method by name update to receive insert stream events row-by- row • The number and types of parameters declared by the update method must match the number and types of columns as specified in the select clause, in the same order as in the select clause • JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 38. Subscriber select orderId, price, count(*) from OrderEvent public class MySubscriber { ... public void update(String orderId, double price, long count) {...} } JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 39. Esper Engine Integration • Obtain an Esper engine instance • Create a statement (using the Esper query language) • Register the statement with the engine • Create a listener (by implementing a Java interface that will be triggered when the statement evaluates to true) and attach it to the statement JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 40. Esper Engine Integration //Register events Configuration configuration = new Configuration(); configuration.addEventType("TempEvent", BodyTemperatureEvent.class.getName()); // Get Esper CEP Engine EPServiceProvider cepEngine = EPServiceProviderManager.getProvider("esper-sports- poc", configuration); // Create EPStatement PlayerBMIStatement playerBMIStatement = new PlayerBMIStatement(cepEngine.getEPAdministrator()); playerBMIStatement.addListener(new PlayerBMIListener()); // Dispatch Event cepEngine.getEPRuntime().sendEvent(eventInstance); JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 41. EPL Similar to SQL Stream Name BP Event Ross 110 Rachel 120 Joe 100 JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 42. Sliding Window Support JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 43. Streams Input stream denotes new events arriving, the term remove stream denotes events leaving a data window. JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 44. Windows • win:length(4) • win:time(4 sec) • win:time_batch(4 sec) • win:length_batch(4) JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 45. Batch Time Window JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 46. Named Windows • Analogous to database views • Created using SELECT queries • Define window (view) first and insert data create window OrdersTimeWindow.win:time(30 sec) as select symbol, volume, price from OrderEvent JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 47. Filters select * from Withdrawal(amount>=200).win:le ngth(5) JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 48. Aggregations • sum • avg • count • min/max • stddev JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 49. Output Rate Limiting Engine dispatches to Listener when output condition occurs. e.g. select irstream symbol, volume, price from MarketData.win:time(5.5 sec) output every 1 seconds JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 50. Event Patterns Pattern Expression = Pattern atoms + Pattern Operators every (spike=ServiceMeasurement(latency>20000) or error=ServiceMeasurement(success=false)) select s from pattern [every s=SubmitOrderEvent -> (timer:interval(30 sec) and not OrderAcceptedEvent(theOrder.id=s.theOrder.id))] JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 51. RDBMS Integration JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 52. RDBMS Integration select name, healthy_temp from TempEvent.win:time(60 sec) temp, sql:mydb3 [' select name,healthy_temp from Player where name = ${temp.playerName} and $ {temp.temperature} > healthy_temp '] JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 53. Non Event Based Data Integration public class InstrumentRepository { public static Exchange[] getExchangesForSymbol(String symbol) { ... } select ord, eligibleExchanges from TheOrder as ord, method:com.ociweb.jnb.esper.InstrumentRepository.ge tExchangesForSymbol(ord.symbol) as eligibleExchanges } JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 54. CEP Design Patterns (1/9) Filtering: e.g. Click-stream analysis: a filter may be used to capture the trades that originate from a certain set of IP addresses. JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 55. CEP Design Patterns (2/9) In Memory Caching: e.g. Click-stream analysis: a typical application may hold the recent clicks and searches performed by the users, coupled with the relevant historical and reference information. JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 56. CEP Design Patterns (3/9) Aggregation over Windows: e.g. Click-stream analysis: it is often useful to compute the number of visitors who click on a particular link within a specified time interval. JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 57. CEP Design Patterns (4/9) Database Lookups: e.g. Trading: a trading application may look up historical price for a stock, or certain information about an order, or certain rules and regulations stored in a database. JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 58. CEP Design Patterns (5/9) Database Writes: e.g. Click-stream analysis: storing the raw click-stream history, together with derived data, in the database JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 59. CEP Design Patterns (6/9) Correlation (Joins): e.g. Trading: correlating information from multiple exchanges to find arbitrage opportunities. JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 60. CEP Design Patterns (7/9) Event Pattern Matching: e.g. Fraud detection: fraud patterns are often described as a sequence of events, in one or more streams. JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 61. CEP Design Patterns (8/9) Hierarchical Events: e.g. Order Processing: Purchase Order can have multiple items which needs to be processed differently. JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 62. CEP Design Patterns (9/9) Dynamic Queries: e.g. Trading environments: every trader can enter their subscriptions like X=MSFT, Y=.1%, Z=10 am. JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 63. EsperIO Provides adapters support: • CSVAdapter • JMSAdapter • OpenTick Adapter JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 64. Demo JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 65. Performance Tips // Better performance select * from RFIDEvent // Less good performance select assetId, zone, xlocation, ylocation from RFIDEvent JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 66. Conclusion Do I really need it? Do I have the right infrastructure? JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 67. Conclusion • Learning Curve of EPL • No Standardization of EPL • EPL is still evolving JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 68. Questions? JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 69. CEP in Dutch Market • KLM - TIBCO • Boekhandels Groep Nederlands (BGN) JTeam Innovating J2EE Development Wednesday, April 1, 2009
  • 70. Resources Esper http://esper.codehaus.org/ http://complexevents.com/ CEP http://www.cepbusinesscast.com/ Open Source SOA JTeam Innovating J2EE Development Wednesday, April 1, 2009