EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

epts
event processing technical society

Patterns v2.04b

epts
Content by: members of the
EPTS Reference Architecture Working Group

Adrian Paschke
Event Processing (Freie Universitaet Berlin)
Paul Vincent
Reference Architecture (TIBCO Software)
Catherine Moxey, Martin Hirzel,
Dave Rushall, Richard Jacks (IBM)
- Design Patterns Alex Alves
(Oracle)

epts

Agenda

• EPTS Reference Architecture Introduction
• EP Patterns:
– EPTS RA patterns
– Some Use Cases
– EPIA patterns

2

epts

Introduction to Event Processing

Event Processing,
Complex Event Processing,
Event Stream Processing
Event Processing Rules, etc.
Event Type Definitions,

Event Abstraction, Event Pattern
Detection, Event Composition

Event Management
etc. etc.

Events Derived
Events

Event Producer Event Consumer
(Event Source, (Event sink, event
Event Emitter) handler, event
listener)

3 Design time Run time Administration

epts

Reference Architecture: Functional View
Definition, Modeling, (continuous) Improvement

Event Reaction
Assessment, Routing, Prediction,
Discovery, Learning
(Pattern, Control, Rule, Query, RegEx.etc)

Event Process Monitoring, Control
0..*

State Management
Complex Event Detection
Event and Complex Event

Consolidation, Composition,
Aggregation
0..*
Event Analysis
Analytics, Transforms, Tracking,
Scoring, Rating, Classification
0..*
Event Preparation
Identification, Selection, Filtering,
Monitoring, Enrichment
0..*

Event Production Event Consumption
Publication, Dashboard, Apps,
Retrieval External Reaction

4
Design time Run time Administration

epts

Comparison of EPTS RA Fns and EPIA EP Agents
Event Reaction
Discovery, Learning

Aggregation

Event Analysis
Analytics, Transforms, Tracking, EPA

Event Preparation Filter Transformation Pattern detect

Translate Aggregate Split Compose

Enrich Project

5

epts

Agenda

• EP Meta Pattern / Pattern Structure

6

epts

What do we want to cover in an EP Pattern?

Type / Operands Operation /
synonym Operators

Event Filtering: a stream or list of events may be filtered
on some payload or metadata information
such that some subset is selected for further processing.

Rationale /
Mapping to business rule
EPL template

7

epts

EP MetaPattern / Pattern Description (version 1)

• Name & Alternative Names
• Classification versus other references
– EPTS RA and other references e.g. “EP in Action” book
• Description
– Role, and Business Rule type specification / requirement
• Structure
– EPTS Glossary terms
• Implementations

8

epts

EP MetaPattern future versions?

• Name & Alternative Names
• Classification versus other references
– EPTS RA and other references e.g. “EP in Action” book
• Description may revise/improve EP Fns
classification / ontology
– Role, and Business Rule type specification / requirement
• Structure
compare/extend
– EPTS Glossary terms other “event patterns”
• Implementations
Extend examples
pseudocode – real example
9

epts

EP Pattern Implementations
• Differ in Implementation type (Event Processing
Language, Execution semantics, etc)
• Sampled systems we cover:
1.TIBCO BusinessEvents Rete-production-rule-engine
(with continuous query and event-expression options)
2.Oracle Event Processing Stream-processing engine
3.Prova Logic Programming Semantic CEP Rule Engine
4.IBM Decision Server Events component of WebSphere
Operational Decision Management
& IBM InfoSphere Streams stream processing engine
10

epts

Implementations:Sampled Systems:
TIBCO BusinessEvents
• TIBCO BusinessEvents “event server” / event processing engines
• Multiple languages / engines in various packaging options
– UML-based event and concept (object) class hierarchical models
– Java-based language
– Eclipse-based IDE
• Distributed execution model:
– Inference agents, Query agents, Datagrid agents
– In-memory, memory+grid, explicit grid operational modes
– “Grid Options” of TIBCO ActiveSpaces / Oracle Coherence
– eXtreme Event Processing use cases: Fedex, US Govt, …
• Transactional model option:
– TIBCO AS Transactions
– eXtreme Event Processing use cases: Telco
• Extensible: eg OWL import, RDF import, etc

epts

TIBCO BusinessEvents (ctd)

Events Event and Data
Structures
Information

Inference Rules

States and
Transitions Continuously
process events
Sets and using procedural
Queries Stored and declarative
Events and
Temporal Data event processing
Patterns elements
Decision Tables

via JMS, RV, MQ, Event Event
TCP/IP, etc… Processing Storage

epts

Oracle Event Processing (formerly Oracle CEP)
• Development platform for event processing applications
• Application model based on EPN (event processing network)
abstraction running on top of OSGi-based Java container.
• Language is an extension of SQL with stream and pattern matching
extensions

13

epts

Oracle Event Processing (formerly Oracle CEP)

• CQL: Continuous Query Language
• Leverages SQL, extended with Stream, Pattern-matching, and Java.
• Continuous: driven by time, and events
• Highly-sophisticated push-down technology to RDBMS, Cache, Hadoop

14

epts

Prova (http://prova.ws) Strong Coupling
direct API
implementation
• Java JVM based, open source rule Prova
language for reactive agents and event Computer

processing
• Leverages declarative ISO Prolog standard Loose Coupling Agent / Service
Interface

extended with (event,message) reaction Message Interchange
Prova
logic, type systems (Java, Ontologies), Agent
Request
query built-ins, dynamic Java integration. Rule Service 0.0000 …
Consumer
• Combines declarative, imperative (object- 0.0000
Response
Prova
Agent
oriented) and functional programming
styles
Decoupled
• Designed to work in distributed Enterprise Prova Event
Service Bus and OSGi environments
Event Streams Processing
(ordered events) Network
• Supports strong, loose and decoupled logout
new auto pay account login
deposit withdrawalnew auto pay Prova
Event
transfer Prova
book logout
interaction deposit
enquiry
account login Agent Agent

• Compatible with rule interchange standards Cloud
account balance
incident request
new auto pay account login
enquiry

enquiry
(unordered events)
depositwithdrawalnew auto pay
such as Reaction RuleML transfer
enquiry deposit logout
account login Prova
activity history Agent Prova
15 Agent

epts

IBM WODM Decision Server Events
• Decision Server Events component of IBM WebSphere Operational
Decision Management (WODM)
• Manages business events flowing across systems and people to provide
timely insight and responses
• Detect, evaluate, and respond to events
• Discover event patterns and initiate actions

16

epts

IBM CICS Transaction Server for z/OS
CICS TS

Event Reaction
Discovery, Learning

Aggregation

Event Analysis

Event Preparation
Identification, Selection, Filtering, Monitoring,
Enrichment

Event Production
Publication, Retrieva, Event Consumption
Identification, Dashboard, Apps,
Selection, Filtering, External Reaction
Enrichment,
Transforms, Routing
17

epts

IBM InfoSphere Streams – Stream Processing Language
• IBM InfoSphere Streams: platform for analyzing big data in motion i.e.
high-volume, continuous data streamsSPL
• IBM Streams Processing Language (SPL)
– Programming language for InfoSphere Streams
– Exposes a simple graph-of-operators view
– Provides a powerful code-generation interface to C++ and Java

Streams Processing Runtime Tools and
Language and IDE Environment Technology
Integration

Front
Office 3.0

Streams Console & Monitoring,
Streams Studio Highly Scalable
Built-in Stream Relational Analytics,
Eclipse IDE for SPL stream processing runtime
Adapters, Toolkits
18

epts

Agenda

• Functional Reference Architecture-based
EP Patterns

19

epts

Patterns Coverage

Event Reaction
Discovery, Learning

State Management
0..*
Aggregation
0..*
Event Analysis
0..*
Event Preparation
0..*

Event Production Event Consumption
Application Time Dashboard, Apps,
Publication, Retrieval External Reaction
20

epts

EP Patterns:RA:Preparation
Event Preparation

Event preparation is the process of preparing the event
and associated payload and metadata for further stages
of event processing.

For example, event preparation may involve the separation
and discarding of unused event payload data,
and the reformatting of that payload data
for downstream event processing.

21

epts

RA:Preparation:Identification:

Event Preparation

Identification: incoming events identified relative to prior
events and event types
– E.g. associating a SMS event with particular mobile phone
account
– E.g. recognizing an event from data (event entity recognition,
extraction) and identifying an event to be of a particular event
type

22

Event Preparation
epts Identification, Selection,
event processing technical society Filtering, Monitoring, Enrichment

Preparation:Identification:Classification

• Alternative Names:
– Event Lookup, Event Entity Recognition, Event Extraction
• EPTS Reference Architecture:
– “Incoming events will need to be identified relative to prior
events and event types, such as associating events with
particular sources or sensors or recognizing / extracting
events from data relative to event type information (event
type systems, event ontologies)"
• EPIA:
– Transform, Translate, Enrich / Project

23

Event Preparation

Preparation:Identification:Description

• Role / Explanation
– Associating an event with some existing entity (data or past
event, event type)
• Associated Business Rule Specification
– a selection (to be enforced during the processing of events):
All <event entities> whose <attribute1> matches the
<attribute2> of <some other event or data or type>
is related by <relationship> to that <some other event or
data or type>. Other data
or event or type

Identification known
event event
24
f(event)

Event Preparation

Preparation:Identification:Structure

• EPTS Glossary comparison
– Event type (event class, event definition, or event
schema)
• A class of event objects.
• Event types should be defined within some type definition
system ... will usually specify certain predefined data
(attributes), examples of which might be:
– A unique event identifier used to reference the event

25

Event Preparation

Preparation:Identification:Implementations

• TIBCO, Prova, Oracle and IBM implementations for
this pattern

Other data
or event

Identification known
event
f(event) event

26

Event Preparation

Preparation:Identification:Implementations:
TIBCO BusinessEvents: Rules
• General pattern: Entity
<declare> event, data community
event

<if> event-data-match Identify
association
<then> create-event-
association-with-data Add
association
• Effect: find appropriate data that to event
matches the incoming event*
and create a reference between
Event
the event and the data
* note: event object = concept: input events are immutable**
27
** note: except in PreProcessor 

Event Preparation

Oracle EP: queries

• General pattern: event
SELECT <property>
FROM <SOURCE> SOURCE
WHERE <predicate-condition>

event

• Effect: matches events coming from Event-type
<SOURCE> with pre-defined event-type

28

Event Preparation

Preparation:Identification:Implementations:Prova
Semantic Identification
• General pattern:
<rcvMsg / rcvMult> Event ^^ Semantic Type :-

Semantic
Knowledge Base

Event Semantic Semantic
Identification
Data f(event) event

• Effect: find appropriate event entity type that matches the
incoming event data and create a reference between the event
and the event type (from the semantic knowledge), so that the
syntactic (event) data becomes a (semantic) event, which has
relations with the semantic background knowledge base
29

Event Preparation

Preparation:Semantic Identification:Prova
Semantic Rules
Event Stream
{(Name, “OPEL”)(Price, 45)(Volume, 2000)(Time, 1) }
{(Name, “SAP”)(Price, 65)(Volume, 1000) (Time, 2)}

rcvMult(SID,stream,“S&P500“, inform,
tick(Name^^car:Major_corporation,P^^currency:Dollar,
T^^time:Timepoint)) :- …<further processing of semantic event>.
Identify event and associate with
semantic background knowledge
{(OPEL, is_a, car_manufacturer),
(car_manufacturer, build, Cars),
(Cars, are_build_from, Metall),
Semantic (OPEL, hat_production_facilities_in, Germany),
Knowledge Base (Germany, is_in, Europe)
(OPEL, is_a, Major_corporation),
(Major_corporation, have, over_10,000_employees)}
30

Event Preparation

• Example: event rule relates to ‘new purchase order’ events,
and the events are processed in the context of the customer
number

• Effect: presence of ‘new purchase order’ in the rule
identifies that event type, and the events are processed in
the context of the customer number field – the events of
interest could be said to be identified by ‘new purchase
order’ type and customer number, or the latter could be
regarded as selection
31

Event Preparation

IBM InfoSphere Streams (SPL)
• Example:
type Kind = enum { knownKind1, knownKind2, knownKind3 };
/*...*/
stream<rstring key, int32 payload> Events = PrepId_EventsSource() {}
stream<rstring key, Kind kind> Mappings = PrepId_MappingsSource() {}
stream<Kind kind, int32 payload> KnownEvents = Custom(Events; Mappings) {
logic
state:
mutable map<rstring, Kind> _keyToKind = {};
onTuple Events:
if (key in _keyToKind)
submit({ kind=_keyToKind[key], payload=payload }, KnownEvents);
onTuple Mappings:
_keyToKind[key] = kind;
}
stream
Events Custom Known
function Events
Mapping stream Filter to map Event Identified
32 key to known kind known events

epts

RA:Preparation:Selection:

Event Preparation

Selection: particular events selected for further analysis or
pattern matching
– E.g. Selecting the n’th event as a part of a sampling function
– E.g. Selecting events related to some existing event to allow
enrichment of that existing event

33

Event Preparation
epts Identification, Selection, Filtering,
event processing technical society Monitoring, Enrichment

Preparation:Selection:Classification

– Event Query
– See also Event Filtering and Event Identification
– “During event preparation, particular events may be
selected for further analysis. Different parts of event
processing may require different selections of events.
See also event filtering. "
• EPIA:
– Filter
34

Event Preparation

Preparation:Selection:Description

– Selecting or locating an event due to some criteria or rules
<attribute2> of <some other event or data>
is selected for <processing>.

Other data
or event

Selection selected
event
35
f(event) event

Event Preparation

Preparation:Selection:Structure

– Event type (event class, event definition, or event
schema)
• A class of event objects.
• Event types should be defined within some type definition
system ... will usually specify certain predefined data
(attributes), examples of which might be:
– A unique event identifier used to reference the event

36

Event Preparation

Preparation:Selection:Implementations

this pattern

Other data
or event

Selection selected
event
f(event) event

37

Event Preparation

Preparation:Selection:Implementations:
<declare> event, data community
event

<if> event-data-match Compare event
vs selection
<then> process-event criteria

• Effect: filter event per selection criteria Event
(Note: action may be to
mark event as selected)

38

Event Preparation

Preparation:Selection:Implementations:Prova
<rcvMsg / rcvMult>
(CID,Protocol, Agent,Performativ,Event|Context)
:- <selection_processing> Event. Existing Event, Data,
Guards, Rules
Event
Selected Compare Event vs.
Selection
selection criteria
f(message(event)) Event
Event ((Events, Data, Guards)

• Effect: select event according to message context (conversation
id, protocol, agent, pragmatic performativ, event context) (see
also monitoring) and further process selected events, e.g.
comparing them against existing events, data, guards, rules,
39
39
selection functions

Event Preparation

Messaging Reaction Rules in Prova
• Send a message
sendMsg(CID,Protocol,Agent,Performative,[Predicate|Args]|Context)
• Receive a message
rcvMsg(CID,Protocol,Agent,Performative,[Predicate|Args]|Context)
• Receive multiple messages
rcvMult(CID,Protocol,Agent,Performative,[Predicate|Args]|Context)

Selection according to:
– CID is the conversation identifier
– Protocol: protocol e.g. self, jms, esb etc.
– Agent: denotes the target or sender of the message
– Performative: pragmatic context, e.g. FIPA ACL
– [Predicate|Args]|Context or Predicate(Arg1,..,Argn): Event
40

Event Preparation

Example
% Select stock ticker events from stream "S&P500"
% Each received event starts a new subconversation (CID)
which further processes the selected event (select)
rcvMult(CID,stream,“S&P500“, inform, tick(S,P,T)) :-
select(CID,tick(S,P,T)).
% Indefinitely (count=-1) receive further ticker events from
other streams that follow the previous selected event in
event processing group (group=g1). If the price differs
for the same stock at the same time [T1=T2, P1!=P2] then …
select(CID,tick(S,P1,T1)) :-
@group(g1) @count(-1)
rcvMsg(CID,stream, StreamID ,inform, tick(S,P2,T2))
[T1=T2, P1!=P2] %guard: if at same time but different price,
println (["Suspicious:",StreamID, tick(S,P2,T2)]," ").
41

Event Preparation

Oracle EP: queries

SELECT <property>
FROM <SOURCE> select

event

• Effect: select certain properties of events Event
from <SOURCE> properties

42

Event Preparation


• Uses ‘business objects’ to select properties
of incoming events that will be used when
events are processed
• Business Objects allow for a level of
indirection between incoming events and
specifications made by the business about
how to respond to events
• Mapped via the Event Designer tool

43

Event Preparation

• Example:
type Kind = enum { knownKind1, knownKind2, knownKind3 };

/*...*/

stream<int32 payload> Selected = Functor(KnownEvents) {
param filter : kind == knownKind1;
}

stream<int32 payload> ProcessedEvents = PrepSel_Process(Selected) {}

event selected
stream Functor
events
Select events
Known Kind 1 of
which match
Events known events
44 known kind

epts

RA:Preparation:Filter

Event Preparation

Filter: filter out all events that have some property in their
payload / data
– E.g. customer purchases: filter out those with values < $100

45

Event Preparation

Preparation:Filter:Classification

– "During event preparation, a stream or list of events may
be filtered on some payload or metadata information
such that some subset is selected for further processing."
• EPIA:
– Filter (EPA or Event Processing Agent) performs filtering
only and has no matching or derivation steps, so it does
not transform the input event.

46

Event Preparation

Preparation:Filter:Description

– Comparing some property of the event (an attribute or
metadata) with some other value (from some other event, or
data)
– As a constraint: a selection (to be enforced during the
processing of events):
All <event entities>
that have Filter
<filter expression>
event event
f(event)
must have
<some status>. discarded
event
47

Event Preparation

Preparation:Filter:Structure

– A filter pattern can relate to several terms in the EPTS
Glossary
• can be specified in an event pattern (A template containing
event templates, relational operators and variables...)
• .. by an event pattern constraint (A Boolean condition that
must be satisfied by the events observed in a system...)
• .. as part of an event processing rule (A prescribed method
for processing events.)
• .. or as part of event stream processing (Computing on inputs
that are event streams.)
–
48

Event Preparation

Preparation:Filter:Implementations

• TIBCO, Oracle, Prova and IBM implementations for
this pattern

Filter
event event
f(event)

discarded
event

49

Event Preparation

Preparation:Filter:Implementations:
TIBCO BusinessEvents: rules
<declare> event event

<if> event-boolean-expression Filter
f(event)

<then> update-event-with-fact event
OR
action-for-filtered-event
Operation on
• Effect: ignore filtered events event
on operations on non-filtered events

50

Event Preparation

TIBCO BusinessEvents: event removal rules
• Implementation type: Rete-based rule-driven system
• Alternate pattern:
<declare> event
event
<if>
NOT Filter
NOT event-boolean-expression
!f(event)
<then>
consume-event event

• Effect: removed filtered events from
system (and ALL subsequent operations!)

51

Event Preparation

TIBCO BusinessEvents: State transitions
<state models for concept C> event
<route from State n to n+1>
<on> event Filter
<where> event-filter query(event)

• Effect: change state for some concept State
transition
• Notes: this is a specialisation of the
general rule pattern!

52

Event Preparation

TIBCO BusinessEvents: Queries
<select> output event
<from> event
<where> event-filter Filter
query(event)

<policy> notification Policy Event
event subset
• Effect: collect filtered events for
event operations per some policy
Operations on
event subset
• Note: the pattern matching framework on policy event
has a similar notion
53 but a different surface syntax

Event Preparation

Oracle EP: queries

SELECT <property>
FROM <SOURCE> predicate

event

• Effect: discard events whose predicate Operation on
evaluate to false event

54

Event Preparation

Oracle EP: model

• Scenario:
– Select only stocks from the stock tick stream whose
symbols are ‘AAA’.
• Application Model

55

Event Preparation

Oracle EP: query

Specify STREAM source

SELECT *
FROM StockTickStream
Define predicate for
WHERE symbol = ‘AAA’ filtering

56

Event Preparation

Prova: rules
• Implementation type: backward reasoning logical filter
• General pattern template:
<rcvMsg / rcvMult> Event :- event … event
logical filter condition(s),
... .
Event pattern +
logical derivation rule 1 :- logical rule filter
filter conditions.
Backward
logical derivation rule 2 :- Resoning
Rule
event
filter conditions. chaning
...

• Effect: Unify received events with Operation on
event pattern definition and apply logical event
57 filter rules (derivation rules with rule chaining)

Event Preparation

Prova: Messaging Reaction Rules in Prova
• Send a message
sendMsg(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)
rcvMsg(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)
rcvMult(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)

Description:
– XID is the conversation identifier
– Performative: pragmatic context, e.g. FIPA Agent Communication
– [Predicate|Args] or Predicate(Arg1,..,Argn): Message payload
58

Event Preparation

Prova: Example
% Filter for stocks starting with „A“ and price > 100
rcvMult(SID,stream,“S&P500“, inform, tick(S,P,T)) :-
S = "A.*",
P > 100,
sendMsg(SID2,esb,"epa1", inform, happens(tick(S,P),T).

Example with rule chaining
% Filter for stocks starting with „A“ and price > 100
filter(S,P),
sendMsg(SID2,esb,"epa1", inform, happens(tick(S,P),T).
filter(Symbol,Price) :-
Price > 100,
Symbol = "A.*".
59

Event Preparation

Prova: Semantic Event Processing Example
Semantic Query Filer:
Stocks of companies, which have production facilities in Europe and
produce products out of metal and
Have more than 10,000 employees.
Event Stream
{(Name, “OPEL”)(Price, 45)(Volume, 2000)(Time, 1) }
{(Name, “SAP”)(Price, 65)(Volume, 1000) (Time, 2)}

{(OPEL, is_a, car_manufacturer),
(car_manufacturer, build, Cars),
(Cars, are_build_from, Metall),
Semantic (OPEL, hat_production_facilities_in, Germany),
Knowledge Base (Germany, is_in, Europe)
(OPEL, is_a, Major_corporation),
(Major_corporation, have, over_10,000_employees)}

rcvMult(SID,stream,“S&P500“, inform,
tick(Name^^car:Major_corporation,P^^currency:Dollar,
T^^time:Timepoint)) :- … <semantic filter inference> .
60

Event Preparation

• Example:
event event
Define a filter to only process events with a
purchase value greater than £100
Filter

event

• Effect: Only process purchase events
which match the filter, taking the action to Action
send a discount voucher event

61

Event Preparation

application event capture (1)
• Example:
Define a filter to only produce events for banking transactions with event event
a value greater than $5000

Filter

event

Operation on
event

• Effect: Only produce events which match the filtering predicates
62

Event Preparation

application event capture (2) Details
<eventCaptureSpecification>
<name>LargeCashDepositCS</name>
<eventIdentifier>LargeCashDeposit</eventIdentifier>
<description>Capture details for event when large cash deposit is
made</description>
<filter>
<contextFilter>
. . .
</contextFilter>
<locationFilter filterType="PROGRAM_INIT">
<programInit>
<PROGRAM filterValue="CPPEPM22" filterOperator="EQ"
keyword="PROGRAM"/>
<CHANNEL filterValue="" filterOperator="OFF" keyword="CHANNEL"/>
</programInit>
</locationFilter>
<dataFilter>
<filterItem>
<dataFilter filterValue="5000" filterOperator="GT"
languageStructureFilename="" languageStructureName="" languageVariableName=""
dataPrecision="0" dataType="PACKED" length="8" offset="5" container="APP"PX22"
source="CHANNEL"/>
</filterItem>
</dataFilter>
</filter>
<dataCapture>
. . .
</dataCapture>
</eventCaptureSpecification>

63

Event Preparation

• Example:
stream<rstring name, uint32 age> People = PrepFil_Source() {}

stream<rstring name, uint32 age> YoungPeople = Filter(People) {
param filter : age < 30u;
}

event event
stream Filter
stream
Filter age < 30
People Young People

event
stream
64
Un-Young People

epts

RA:Preparation:Monitoring:

Event Preparation

Monitoring: particular event channels are monitored to identify
events of interest
– E.g. Selecting a particular channel and event selector in
some middleware subscription

65

Event Preparation

Preparation:Monitoring:Classification

– Event Subscribe
– See also Event Selection
– “During event preparation, particular types of events may
be monitored for selection for further processing. This
may utilise specific mechanisms external to the event
processing such as exploiting event production features."
• EPIA:
– Filter
66

Event Preparation

Preparation:Monitoring:Description

– Directing an external agent to select events
– Applying some monitor function over some event channel
– a monitor (of an event source):
<attribute2> of <some monitoring criteria>
is selected for <processing>.
Monitor
criteria

Monitor monitored
event event
67
f(event)

Event Preparation

Preparation:Monitoring:Structure

– (Example mechanism): Publish-and-subscribe (pub-sub)
• A method of communication in which messages are
delivered according to subscriptions .
• Note 1. Subscriptions define which messages should flow to
which consumers.
• Note 2. Event processing applications may use publish-and
- subscribe communication for delivering events. However,
publish - and - subscribe is not definitional to event
processing – other communication styles may be used.
– Subscriber
• An agent that submits a subscription for publish- and –
subscribe communication.
68

Event Preparation

Preparation:Monitoring:Implementations

this pattern

Monitor
criteria

Monitor monitored
event event
f(event)

69

Event Preparation

Preparation:Monitoring:Implementations:
TIBCO BusinessEvents: Event Destination
<define> channel, channel
destination, event
Channel
Destination
• Effect: subscribe to events on this
channel per this destination
(Note: part of Event definition) Event

70

Event Preparation

Oracle EP: Event Destination
<channel> channel

<name>dest1</name>
Channel
<selector>query1</selector> Selector
</channel>
Event

• Effect: subscribe to events on this
channel per selected queries.

71

Event Preparation

Prova: Selection
<rcvMsg / rcvMult>
(CID,Protocol, Agent,Performativ,Event|Context)
:- <selection_processing> Event .

Event
Monitor + Selection Selected
f(message(event)) Event
Event

• Effect: monitor message channels and select (see also
Selection Pattern) events according to message context
(conversation id, protocol, agent, pragmatic performativ, event
72
context);
72

Event Preparation

Prova: Messaging Reaction Rules in Prova
• Send a message
sendMsg(CID,Protocol,Agent,Performative,[Predicate|Args]|Context)
rcvMsg(CID,Protocol,Agent,Performative,[Predicate|Args]|Context)
rcvMult(CID,Protocol,Agent,Performative,[Predicate|Args]|Context)

Selection according to:
– CID is the conversation identifier
– Performative: pragmatic context, e.g. FIPA ACL
– [Predicate|Args]|Context or Predicate(Arg1,..,Argn): Event
73

Event Preparation

• Example:
Use an event
connector that
subscribes for
aircraft arrivals
from a JMS topic

• Effect: A message
selector is used to
limit which messages
will trigger events,
and an XSL
stylesheet is used to
transform the
message text into an
event packet.
74

Event Preparation

• Example:
type BargainType = rstring symbol, float32 price;

/* in exporting application */

() as ExportTech = Export(TechSectorBargains) {
param properties : { kind = "bargains", category = "tech" };
}

/* in importing application */

stream<BargainType> TechBargains = Import() {
param subscription : kind == "bargains" && category == "tech";
}

• Effect: Stream of bargains in tech sector is sent (exported) from
current application, making it available to another application which
imports the stream to monitor bargains in various sectors
75

epts

RA:Preparation:Enrichment:

Event Preparation

Enrichment: add some information based on prior events
– E.g. customer purchase: add the customer history to the
purchase event
– E.g. enrich the event with semantic background knowledge

76

Event Preparation

Preparation:Enrichment:Classification

– Event Annotation, Semantic Event
– "During event preparation, events may be "enriched"
through knowledge gained through previous events or
data (including semantic background knowledge). "
• EPIA:
– Enrich (EPA): a translate EPA that takes a single input
event, uses it to query data from a global state element,
and creates a derived event which includes the attributes
from the original event, possibly with modified values...
77

Event Preparation

Preparation:Enrichment:Description

– Updating some attribute or metadata of the event with some
other values, possibly computed, from some other event or
data
All <event entities> that are also <enriched data
expression>
Other data
must have <some status>. or event

Enrichment
event event
f(event)
78

Event Preparation

Preparation:Enrichment:Structure

– Enrichment can be specified in an event pattern (pattern
definition containing additional meta data, semantic
knowledge)
– .. as part of an event processing rule (A method for
enriching events during processing, e.g., by analysing the
meta data and querying external data sources or
inferencing semantic knowledge bases)
– .. or as part of event stream processing (additional
additional information to event stream data, e.g. by
querying external data sources)

79

Event Preparation

Preparation:Enrichment:Implementations

• TIBCO, Oracle, Prova and IBM implementations for
this pattern

Other data
or event

Enrichment
event event
f(event)

80

Event Preparation

Preparation:Enrichment:Implementations:
• General pattern: Enrich
<declare> event, data data
event

<if> event-data-match Identify
enrichment data

<then> update-event-with-data Enrich
Operation
• Effect: find appropriate data for on event
events and apply enrichment
operations on them*
Event

81
* note: event object: events cannot be modified

Event Preparation

Preparation:Enrichment:Implementations: Oracle
EP: queries
event
SELECT <property>
FROM <STREAM>[WIN],<RELATION>
join-condition
WHERE <join-condition>

event
• Effect: enrich events from stream
with contextual data from relation
where condition applies. Operation on
event

82

Event Preparation

EP: model

• Scenario:
– For every stock event from stock tick stream whose bid is
greater than 5.0, find its company’s location.
• Application Model

83

Event Preparation

EP: queries

No need for window
SELECT event.symbol, location operator on a ‘relation’
FROM
StockTickStream [NOW] AS event,
StockTable AS data
WHERE event.symbol = data.symbol AND
event.lastBid > 5.0

84

Event Preparation

EP: table metaphor
StockTable
Symbol Full-name Location
AAA The AAA company San Francisco

BBB The BBB company San Jose

Time Input (StockTickStream) Output (QUERY)
1 {“AAA”, 10.0, 10.5} {“AAA”, “San…”}

2 {“BBB”, 11.0, 12.5} {“BBB”, “San Jose”}

3 {“AAA”, 4.0, 4.5}

85

Event Preparation

Preparation:Enrichment:Implementations: Prova:
rules
rcvMsg / rcvMult <Event Msg Pattern(s)> :- event
<query built-in(s)>,
<enrichment condition(s)> External
... . Query data
<logical enrichment rule 1> :-
<enrichment operations>. Enrichment
rules

• Effect: query data or knowledge from event*
external source and enrich the event with it
Operation on
enriched event
86

Event Preparation

Preparation:Enrichment:Implementations: Prova:
External Data and Object Integration
• Java
..., L=java.util.ArrayList(),, ..., java.lang.String(“Hello”)

• File Input / Output
..., fopen(File,Reader), ...

• XML (DOM)
document(DomTree,DocumentReader) :-
XML(DocumenReader),
• SQL
…,sql_select(DB,cla,[pdb_id,“1alx“],[px,Domain]).
• RDF
...,rdf(http://...,"rdfs",Subject,"rdf_type","gene1_Gene"),
• XQuery
..., XQuery = ' for $name in
StatisticsURL//Author[0]/@name/text() return $name',
xquery_select(XQuery,name(ExpertName)),
• SPARQL
...,sparql_select(SparqlQuery,name(Name),class(Class),
87 definition(Def)),

Event Preparation

Prova: Example with SPARQL Query
% Filter for car manufacturer stocks and enrich the stock tick
event with data from Wikipedia (DBPedia) about the manufacturer
and the luxury cars
carManufacturer(S,Man), % filter car manufacturers
findall([luxuryCar|Data],luxuryCar(Man,Name,Car),Data), % query
EnrichedData = [S,Data], % enrich with additional data
sendMsg(SID2,esb,"epa1", inform, happens(tick(EnrichedData,P),T).

% rule implementing the query on DBPedia using SPARQL query
luxuryCar(Manufacturer,Name,Car) :-
Query="SELECT ?manufacturer ?name ?car % SPARQL RDF Query
WHERE {?car <http://purl.org/dc/terms/subject>
<http://dbpedia.org/resource/Category:Luxury_vehicles> .
?car foaf:name ?name .
?car dbo:manufacturer ?man .
?man foaf:name ?manufacturer.
} ORDER by ?manufacturer ?name“,
sparql_select(Query,manufacturer(Manufacturer),name(Name),car(Car)).
88

Event Preparation

• Example: Define a Mapped Key expression, where a event
field in the event is used to look up objects in a
database table
Mapped Key
look up

Enriched
event

• Effect: Required enrichment data is obtained from the
data base and added to the event

89

Event Preparation

• Example:
/* VWAP = volume-weighted average price */
stream<rstring sym, float32 vwap> Vwaps = PrepEnr_VwapsSource() {}

stream<rstring sym, float32 price> Trades = PrepEnr_TradesSource() {}

stream<rstring sym, float32 vwap, float32 price> E = Join(Vwaps; Trades) {
window
Vwaps : sliding, count(0);
Trades : sliding, count(100), count(1), partitioned;
param
equalityLHS : Vwaps.sym;
equalityRHS : Trades.sym;
partitionByRHS : Trades.sym;
}
stream
Trades Enriched Trades
Join enriched with
Stream
VWAPs
stream VWAPs

90

epts

EP Patterns:RA:Analysis

Event Analysis

Event Analysis is the process of analysing suitably
prepared events and their payloads and metadata for
useful information.
For example, event analysis may involve the
identification of events against existing entities.

91

epts

RA:Analysis:Analytics

Event Analysis

Analytics: use of mathematical methods like statistics to
derive additional information about an event or set of
events
– E.g. standard deviation of the response time of a system,
used to give a quantitative comparison of current with past
performance of a system

92

Event Analysis
epts Analytics, Transforms, Tracking,
event processing technical society Scoring, Rating, Classification

Analysis:Analytics:Classification

– Event Statistics
– “Event Analytics are the use of statistical methods to
derive additional information about an event or set of
events.”
• EPIA:
– Aggregate (EPA): a transformation EPA that takes as
input a collection of events and creates a single derived
event by applying a function over the input events.

93

Event Analysis

Analysis:Analytics:Description
– Analytics is where multiple data items (in a single event, or
more commonly across multiple events), are analysed
mathematically to define some additional information
– For example: an average or mean value of event data can
be used to compare against subsequent events to indicate
some trend (see Learning)
– an event analytics (event processing) process:
The <analytic fact> is provided by <analytic function>
applied to <events>
Analytics
event info
event f(events)
94
event

Event Analysis

Analysis:Analytics:Structure

– Cause: An event A is a cause of another event B, if A had to happen
in order for B to happen.
– Event abstraction: The relationship between a complex event and the
other events that it denotes, summarizes, or otherwise represents.
– Derived event (synthesized event, synthetic event): An event that is
generated as a result of applying a method or process to one or
more other events.
– Relationships between events: Events are related by time,
causality, abstraction, and other relationships. Time and causality
impose partial orderings upon events.
– Event pattern: A template containing event templates, relational
operators and variables.
– Constraint (event pattern constraint): A Boolean condition that must
95 be satisfied by the events observed in a system.

Event Analysis

Analysis:Analytics:Implementations

• TIBCO, Prova, Oracle, IBM implementations for this
pattern

past
event data

Analytics
event info
f(events,data)

96

Event Analysis

Analysis:Analytics:Implementations:
<declare> event,
past data event
historic-event-object
<if> Statistical
event-matches-statistical- operation
requirement or process
<then>
apply-update-function for Information
event, historical-event-object

• Effect: event-driven statistic function (eg update of mean
for a given event type or other condition)
97

Event Analysis

Prova: Rules
(past) data
rcvMsg / rcvMult <Event Msg Pattern(s)> :-
<query (past) data>,
<analytics functions(s)
with data> Analytics
... . function(s)

<analytics rule 1> :-
<Java call to analytics API>,
Information
<or Prova functional programming>.

• Effect: event-driven statistic function (eg update of mean
for a given event type or other condition)
98

Event Analysis

Prova: Java Programming and Functional Programming
Prova Dynamic Java Programming

• Ability to embed Java calls
… :- X = java.lang.Math.abs(2.99), X+1, …
• Java constructors, object instance + methods and static methods, and
public field access; (+ Java exception handling)

Prova Functional Programming

• single‐ and multi‐valued functions,
• functional composition with the extended derive built‐in;
• partial evaluation;
• lambda functions;
• monadic functions;
• monadic bind using a composition of map and join;
• maybe, list, state, tree and fact monads
99

Event Analysis

Oracle EP: queries

SELECT analytical-
function(<property>)
analytics
FROM <SOURCE>
info

• Effect: performs analytical function on
selected properties, such as std-dev,
regressions, etc.

100

Event Analysis


• General Pattern: Filters in Decision Server
Events allow analytic expressions to be applied

101

Event Analysis

• Example:
stream<rstring sym, float32 vwap, float32 price> E = Join(Vwaps; Trades) {
/* (see Preparation:Enrichment) */
}

/* bargain if price lower than volume-weighted average price */
stream<rstring sym, float32 index> Bargains = Functor(E) {
output Bargains : index = (vwap <= price)
? 0f : price * exp(vwap – price);
}

event event
stream Functor
stream
Analyze trades to
Trades Bargains
with ave price detect bargains

102

epts

RA:Analysis:Transforms

Event Analysis

Transforms: processes carried out on events’ payloads or
data, such for as standardising schema or information
types
– E.g. an insurance application event from a custom IT system
may have the customer information translated into an
ACORD-type structure for onward processing

103

Event Analysis

Analysis:Transforms:Classification

– Event Processes, Event Operations
– “Event Transforms are processes carried out on event
payloads or data, either related to event preparation,
analysis or processing.”
• EPIA:
– Transformation (EPA): an EPA that includes a derivation
step and optionally also a filtering step.
• Derived event: an event generated as a result of EP in EPA

104

EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (20)

Similar to EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b

Similar to EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b (20)

Recently uploaded

Recently uploaded (20)

EPTS DEBS2012 Event Processing Reference Architecture - Design Patterns v2_04b