Complex Event Processing with Esper


Published on

Nathan Reese's presentation to DOSUG on Esper's complex event processing engine.

Published in: Education, Technology, Sports
  • I see, and thank you for your help, online should be good enough.
    Are you sure you want to  Yes  No
    Your message goes here
  • Wingchin: Sorry, I didn't have authorization from this presenter to make them available for download, only online. But I will ask him again.
    Are you sure you want to  Yes  No
    Your message goes here
  • I want to download!
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Distributed systems span across the boundaries between enterprises Examples Financial trading systems Same structure: dispersed set of applications communicating with one another by messages transmitted over an IT layer
  • Distributed Information Systems aka global communications spaghetti pot Activities are driven by events flowing through IT layers No technology to enable humans to understand what is going on Tools to see events No tools to make sense of events
  • Need tools to unravel the communication spaghetti, to be able to understand it at a human level and derive instant insights. Need to be able to determine the following from the event cloud
  • New suite of technologies have been created to answer these questions
  • A good way to visualize ESP and CEP is to think of an upside down database. A traditional databases or distributed caches are passive data structures that create frozen and slow assets which require explicit querying to make sense of.
  • DB model turned upside down. Store your queries and continuously flow data over queries, constantly detecting patterns among events (event correlation), filtering events, aggregate time or length windows of events, join event streams, trigger based on absence of events
  • Real time data mining: Query results returned in real time when conditions occur that match queries Continuous execution model: In CEP, the event stream is constantly being examined for query matches and patterns. Database only searches while a query is running. If your application needs data 10 times per second it must query 10 times per second. Disparate data streams: CEP can have any/all enterprise data streams flowing over queries. Break free from data silos Tiny footprint: Do not need large hardware investment. Benchmark tests running 1000 to 100000 messages per second on a laptop.
  • Not meant to replace traditional databases, which will always be needed for historical data and adhoc queries.  This is meant as another cog in the wheel that will greatly reduce the burden placed on databases while provided much needed real time capabilities.
  • CEP nodes can placed into a hierarchy to model target system’s logical layers
  • Esper engine is an Open Source java CEP library that makes building CEP applications possible Esper is a lightweight kernel written in Java which is fully embeddable into any Java process, JEE application server or Java-based Enterprise Service Bus. It enables rapid development of applications that process large volumes of incoming messages or events. Esper is actively developed with quarterly releases
  • The EPServiceProvider interface represents an engine instance. Each instance of an Esper engine is completely independent of other engine instances and has its own administrative and runtime interface. Create events Set up query statements Specify Engine result consumers (Listener or subscriber) Esper configuration options
  • The Event Processing Language (EPL) is a SQL-like language. Streams (and views) replace tables as the source of data with events replacing rows as the basic unit of data. Views represent the different operations needed to structure data in an event stream and to derive data from an event stream Views define the data available for querying and filtering. Some views represent windows over a stream of events. Other views derive statistics from event properties, group events or handle unique event property values. Lots of built in views
  • Diagram taken from “Esper Reference Documentation” version 3.4 produced by EsperTech
  • Diagram taken from “Esper Reference Documentation” version 3.4 produced by EsperTech
  • Diagram taken from “Esper Reference Documentation” version 3.4 produced by EsperTech
  • Resources and further reading
  • Complex Event Processing with Esper

    1. 1. Complex Event Processing with Esper Turn your database upside down DB Nathan Reese [email_address]
    2. 2. “ Information processing using computer systems on a global scale has become the foundation of the twenty-first-century” David Luckham – The Power of Events
    3. 3. Distributed Information Systems SOA
    4. 4. Global Communications Spaghetti
    5. 5. horizontal/vertical causality detect patterns filter aggregate correlate
    6. 6. Event Driven Architecture (EDA) Complex Event Processing (CEP) Event Stream Processing (ESP)
    7. 7. Store Now Query Later
    8. 8. Query Continuously
    9. 9. <ul><li>Real Time Data Mining </li></ul><ul><li>Continuous Execution Model </li></ul><ul><li>Disparate Data Streams </li></ul><ul><li>Tiny Footprint </li></ul>
    10. 10. <ul><li>Not Database Replacement </li></ul>
    11. 11. Event Streams App1 AppX … CEP
    12. 12. Event Streams App1 AppX … CEP CEP CEP Derived Event Streams
    13. 13. <ul><li>CEP </li></ul>Esper Open Source Java library ESP
    14. 14. Esper Engine Open Source SOA – Jeff Davis EPServiceProvider Event Objects EPL Statements Listeners Subscribers Configuration JDBC Adapter
    15. 15. <ul><li>SQL </li></ul><ul><li>Tables </li></ul><ul><li>Rows </li></ul>EPL Event Streams Views Events
    16. 16. select * from Esper Reference Documentation Ch. 3
    17. 17. select * from Withdrawal(amount>=200).win:length(5) Esper Reference Documentation Ch. 3
    18. 18. select * from where amount >= 200 Esper Reference Documentation Ch. 3
    19. 19. <ul><li>Demo </li></ul>
    20. 21. <ul><li> </li></ul><ul><li> </li></ul><ul><li> </li></ul>
    21. 22. <ul><li> </li></ul><ul><li> </li></ul><ul><li> </li></ul><ul><li> </li></ul><ul><li> </li></ul><ul><li> </li></ul><ul><li> </li></ul><ul><li> </li></ul>