Esper - CEP Engine

Esper
The Power to Predict

JTeam
Innovating J2EE Development
Wednesday, April 1, 2009

Agenda

• Predictive Business
• Complex Event Processing
• Esper
• Demo
• Questions
JTeam

Predictive Business

"In the future, every company must be able to
recognize what is coming early enough to take
evasive action or steer towards an emerging
opportunity."[Frederick W. Smith-CEO
FedEx]

JTeam

Business Demands

High throughput

Complexity High Availability

Low latency Insight

JTeam

Conventional Models

• RDBMS
• Adds delays into data storage
• Uses Polling Model
• Wasteful Use of Resources

JTeam

Conventional Models

• Messaging Systems
• Process Unitary Events
• No Correlation support

JTeam

Complex Event Processing

JTeam

Concepts - Event

An event is a piece of data that represents
that something happened in the real world,
or in software system.
e.g. Stock Tick, Weather, UserClick

JTeam

Concepts - Complex Event

An event that could only happen when lots
of other events happen.
e.g. Housing Price Drop

JTeam

Concepts - Event Stream

A linearly ordered (often by time)
sequence of events forms Event Stream.
e.g. Click Stream

JTeam

Concepts - Event Cloud

A partially ordered set of events form
Event Cloud
e.g.

Set of all stock trades for GOOG within a 5
minute time window is an Event Stream. While all
Stocks sold in a business day is an Event Cloud.

JTeam

Concepts - Event Cloud

JTeam

ESP and CEP

Event Stream Processing is focused more
on high-speed querying of data in streams
of events and applying mathematical
algorithms to the event data.
Complex Event Processing includes event
data analysis, but places emphasis on
patterns of events, and abstracting and
simplifying information in the patterns.
JTeam

CEP Engine

An event processor is an application that
performs operations on event objects,
including creating, reading, transforming,
aggregating, correlating or removing them.

JTeam

CEP Engine

JTeam

CEP Engine Characteristics

1. Keep the data moving
2. Query on streams
3. Handle Stream Imperfections
4. Integrate Stored and Streaming Data
5. Guarantee Data Safety and Availability

JTeam

CEP Engine Characteristics

6. Partition and scale applications automatically
7. Process and Respond instantaneously
8. Interoperability

JTeam

CEP in Application Architecture

JTeam

Commercial CEP Products

• Weblogic Event Server

• Coral8

• TIBCO

• IBM

• StreamBase

• Progress

JTeam

CEP/EDA and SOA [1/3]

EDA is SOA 2.0

EDA is Proactive SOA
EDA and SOA
Complementary

JTeam


JTeam

Real World Applications

Capital Markets
BAM - Business Activity Monitoring
Supply Chain Management
Flight Operations Monitoring
Fraud Detection
JTeam

Esper

• Dual Licensing model
• Open Source GPL

• Esper Tech

• Q2/2007 BEA Partner

JTeam

Good Documentation

• Nice reference document
• Patterns catalogue available online
• Lots of examples

JTeam

Lightweight

• ANTLR
• CGLIB
• LOG4J

JTeam

Ohloh Analysis Summary

JTeam

Pluggability

• Works with any Java Process
• OSGI Enabled

JTeam

Architecture

ESPER Engine

EPServiceProvider

Event EPL
JDB
Objects Statements Confi
C
gurat
Adap
ion
Listeners ter

Subscribers

JTeam

API Overview

• EPServiceProvider
• Acts as Engine
• EPStatement
• Queries written in EQL
• UpdateListener
JTeam

Event

• POJOs
• Key value pairs (java.util.Map)
• XML Documents

JTeam

Event

JTeam

Event Properties

Type Syntax Example
Simple name sensor

Indexed name[index] sensor[0]

Mapped name(key) sensor(‘light’)

Nested name.nested sensor.value
Name
JTeam

UpdateListener

Receives updated data as soon as
it is processed for the statement.

JTeam

Subscriber

• Must provide a method by name update
to receive insert stream events row-by-
row
• The number and types of parameters
declared by the update method must
match the number and types of columns
as speciﬁed in the select clause, in the
same order as in the select clause
• JTeam

Subscriber

select orderId, price, count(*) from
OrderEvent
public class MySubscriber { ...
public void update(String orderId, double
price, long count) {...}
}

JTeam

Esper Engine Integration

• Obtain an Esper engine instance

• Create a statement (using the Esper query
language)

• Register the statement with the engine

• Create a listener (by implementing a Java
interface that will be triggered when the
statement evaluates to true) and attach it to the
statement
JTeam

Esper Engine Integration

//Register events
Configuration configuration = new Configuration();
configuration.addEventType("TempEvent",
BodyTemperatureEvent.class.getName());

// Get Esper CEP Engine
EPServiceProvider cepEngine = EPServiceProviderManager.getProvider("esper-sports-
poc", configuration);
// Create EPStatement
PlayerBMIStatement playerBMIStatement = new
PlayerBMIStatement(cepEngine.getEPAdministrator());
playerBMIStatement.addListener(new PlayerBMIListener());
// Dispatch Event
cepEngine.getEPRuntime().sendEvent(eventInstance);

JTeam

EPL

Similar to SQL
Stream
Name BP
Event
Ross 110
Rachel 120
Joe 100

JTeam

Sliding Window Support

JTeam

Streams

Input stream denotes new events
arriving, the term remove stream
denotes events leaving a data
window.

JTeam

Windows

• win:length(4)

• win:time(4 sec)

• win:time_batch(4 sec)

• win:length_batch(4)

JTeam

Batch Time Window

JTeam

Named Windows

• Analogous to database views
• Created using SELECT queries
• Deﬁne window (view) ﬁrst and
insert data
create window OrdersTimeWindow.win:time(30
sec) as select symbol, volume, price from
OrderEvent
JTeam

Filters

select * from
Withdrawal(amount>=200).win:le
ngth(5)

JTeam

Aggregations

• sum
• avg
• count
• min/max
• stddev

JTeam

Output Rate Limiting

Engine dispatches to Listener
when output condition occurs.
e.g.
select irstream symbol, volume, price from
MarketData.win:time(5.5 sec) output every 1 seconds

JTeam

Event Patterns

Pattern Expression = Pattern
atoms + Pattern Operators
every (spike=ServiceMeasurement(latency>20000)
or error=ServiceMeasurement(success=false))
select s from pattern [every s=SubmitOrderEvent ->
(timer:interval(30 sec) and not
OrderAcceptedEvent(theOrder.id=s.theOrder.id))]

JTeam

RDBMS Integration

JTeam

RDBMS Integration

select name, healthy_temp from
TempEvent.win:time(60 sec) temp, sql:mydb3 ['
select name,healthy_temp from Player where
name = ${temp.playerName} and $
{temp.temperature} > healthy_temp ']

JTeam

Non Event Based Data Integration

public class InstrumentRepository {
public static Exchange[]
getExchangesForSymbol(String symbol) {

...
}
select ord, eligibleExchanges from TheOrder as ord,
method:com.ociweb.jnb.esper.InstrumentRepository.ge
tExchangesForSymbol(ord.symbol) as eligibleExchanges
}
JTeam

CEP Design Patterns (1/9)

Filtering:
e.g. Click-stream analysis: a filter may be used to capture the trades that
originate from a certain set of IP addresses.

JTeam


In Memory Caching:
e.g. Click-stream analysis: a typical application may hold the recent clicks and
searches performed by the users, coupled with the relevant historical and reference
information.

JTeam


Aggregation over Windows:
e.g. Click-stream analysis: it is often useful to compute the number of visitors
who click on a particular link within a specified time interval.

JTeam


Database Lookups:
e.g. Trading: a trading application may look up historical price for a stock, or
certain information about an order, or certain rules and regulations stored in a
database.

JTeam


Database Writes:
e.g. Click-stream analysis: storing the raw click-stream history, together with
derived data, in the database

JTeam


Correlation (Joins):
e.g. Trading: correlating information from multiple exchanges to find arbitrage
opportunities.

JTeam


Event Pattern Matching:
e.g. Fraud detection: fraud patterns are often described as a sequence of
events, in one or more streams.

JTeam


Hierarchical Events:
e.g. Order Processing: Purchase Order can have multiple items which needs to
be processed differently.

JTeam


Dynamic Queries:
e.g. Trading environments: every trader can enter their subscriptions like
X=MSFT, Y=.1%, Z=10 am.

JTeam

EsperIO

Provides adapters support:
• CSVAdapter
• JMSAdapter
• OpenTick Adapter

JTeam

Demo

JTeam

Performance Tips

// Better performance
select * from RFIDEvent

// Less good performance
select assetId, zone, xlocation, ylocation
from RFIDEvent

JTeam

Conclusion

Do I really need it?

Do I have the right
infrastructure?

JTeam

Conclusion

• Learning Curve of EPL
• No Standardization of EPL
• EPL is still evolving

JTeam

Questions?

JTeam

CEP in Dutch Market

• KLM - TIBCO
• Boekhandels Groep Nederlands
(BGN)

JTeam

Resources

Esper http://esper.codehaus.org/
http://complexevents.com/
CEP http://www.cepbusinesscast.com/
Open Source SOA

JTeam

Esper - CEP Engine

More Related Content

What's hot

Similar to Esper - CEP Engine

Recently uploaded

Esper - CEP Engine