OpherEtzion Wed-PM.ppt

IBM Haifa Research Lab – Event Processing
© 2010 IBM Corporation
Event processing – past, present, future
VLDB 2010 Tutorial,
Singapore, September 15th, 2010
Opher Etzion (opher@il.ibm.com)

2
An example to kick-off this tutorial:
Act:
Passenger
has been re-
routed to
another
destination –
send the
luggage
Act: Bag has
reached to
the wrong
aircraft
Notify: Bag
has been
checked but
did not reach
the ULD
within 20
minutes
Notify: Bag
has been
checked but
did not reach
the
connecting
flight
THE LUGGAGE PERSPECTIVE:
Across the 24 largest airlines more
than 5.6 million bags went missing in
2006, this is an average of 15.7 bags
per 1,000 travelers. 15% of the
bags are never found.
BBC News, April 4, 2007
Event processing can
help here..

3
Another example: premature baby monitoring
Patient is hooked up to multiple
monitors (in hospital or at
home) - the physician can set
up event- based rules on
multiple measurements and
patient’s history when to send
an alert and to whom:
Defined
Pattern

4
Outline of this tutorial
What is behind event processing ? How is it
related to other computing terms? Where are its
roots?
Event processing – architecture, building
blocks
The present: State of the practice in event
processing – languages, implementation issues,
challenges in implementing event processing
applications
The future: Trends and research challenges

5
What is behind event processing ? How is it
related to other computing terms? Where are its
roots?

6
What is “event processing” anyway?
or
Event processing is a form of computing that
performs operations on events

7
In computing we ptocessed events since early days
Network and
System
Management

8
Emerging technologies in enterprise computing
(Gartner Hype Cycle, Summer 2009)

9
What’s new? The analog: moving from
files to DBMS
In recent years – architectures, abstractions, and dedicated
commercial products emerge to support functionality that was
traditionally carried out within regular programming.
For some applications it is an improvement in TCO; for others is
breaking the cost-effectiveness barrier.

10
What is an event
There are various definitions of events
Event is an occurrence within a particular system or
domain; the word event has double meaning: the real-world
Occurrence as well as its computerized representation

11
In daily life we often react to events..

12
Sometime we even react to the occurrence
of multiple events
I closed
the deal with the
Canadians
We closed two huge deals in a single day,
It is a good opportunity to send all the team to
Some fun time in Singapore
I closed the deal with the
Australians

13
Event
Patterns
Pattern detection is one of the notable functions of
event processing

14
What we actually want to react to are – situation
TOLL
VILOATOR FRUSTRATED
CUSTOMER
Sometimes the situation is
determined by detecting that
some pattern occurred in the
Flowing events.
Toll violation
Frustrated customer
Sometimes the events can
approximate or indicate with
some certainty that the
situation has occurred

15
Event processing is being used for various
reasons

16
Event Driven Architecture
Event driven architecture: asynchronous, decoupled; each
component is autonomic.

17
Ancestor: active databases
On event
When condition
Do action
With coupling mode
Composite
events were
inherited to
event
processing

18
Ancestor: Data Stream management system
Source: Ankur Jain’s website

19
Ancestor: Temporal databases
There is a substantial temporal
nature to event processing.
Recently – also spatial and
spatio-temporal functions are
being added

20
Ancestor: Messaging – pub/sub middleware

21
Ancestor: Discrete event simulation

22
Ancestor: Network and system management

23
Event processing – architecture, building blocks

24
Flower Store
Van Driver
Ranking and
Reporting
System
Bid Request
Delivery Bid
Assignments,
Bid alerts, Assign
Alerts
Control System
GPS Location
Location Service
Location
Driver’s Guild
Ranking and reports
Delivery
confirmation
Pick Up confirmation
Ranked drivers /
automatic assignment
Bid System
Store
Preferences
Delivery Request
Assignment
System
Manual Assignment
Assignment
Assignments,
Pick Up Alert
Delivery Alert
Fast Flower Delivery

25
Event processing network

26
Example of EPN – part of the FFD example

27
Event Processing Agent
Context
Event
Channel
Event
Consumer
Event
Type
Event
Producer
Global
State
The seven
Building blocks

28
Event type definition
Header Payload Open content
System defined event
attributes
Attributes specific to the
event type
Additional free format
data included in the
event instance

29
Producer – State Observer in workflows
State observer
Push:
Instrumentation points;
Pull:
Query the state

30
Producer – Code instrumentation

31
Producer – syndication

32
Producers – streams to events

33
Producer – sensors

34
Consumer - Performance monitoring dashboard

35
Consumer - Ambient Orb

36
Producer and consumer - Sixth sense

37
Twitter as a consumer

38
Event Processing Agent
Filter Transform Detect Pattern
Translate Aggregate Split Compose
Enrich Project
Event Processing Agents

39
The EPA picture
Output
Not selected
Instance
selection
Context
expression
Pattern detect EPA
Relevance
filtering
Input terminal
filter expression
Relevant event types
Derivation
Derivation
expression
Matching
Pattern signature:
Pattern type
Pattern parameters
Relevant event types
Pattern policies
Pattern
matching set
Participant
events

40
Filter EPA
A filter EPA is an EPA that performs
filtering only, and has no matching or
derivation steps, so it does not
transform the input event.
Filtering
Filter EPA
Filtered Out
Filtered In
Non-Filterable
Principal filter
expression

41
Transform EPA sub types
Translate Compose
Aggregate Enrich
Split
Project

42
Sample of pattern types
all pattern is satisfied when the relevant
event set contains at least one instance of
each event type in the participant set
any pattern is satisfied if the relevant event
set contains an instance of any of the event
types in the participant set
 absence pattern is satisfied when there are
no relevant events
relative N highest values pattern is satisfied
by the events which have the N highest value
of a specific attribute over all the relevant
events, where N is an argument
 value average pattern is satisfied when the
value of a specific attribute, averaged over all
the relevant events, satisfies the value
average threshold assertion.
always pattern is satisfied when all the
relevant events satisfy the always pattern
assertion
sequence pattern is satisfied when
the relevant event set contains at
least one event instance for each
event type in the participant set, and
the order of the event instances is
identical to the order of the event
types in the participant set.
increasing pattern is satisfied by an
attribute A if for all the relevant
events, e1 << e2  e1.A < e2.A
 relative max distance pattern is
satisfied when the maximal distance
between any two relevant events
satisfies the max threshold assertion
moving toward pattern is satisfied
when for any pair of relevant events
e1, e2 we have e1 << e2  the
location of e2 is closer to a certain
object then the location of e1.

43
Pattern detection example
Pattern name: Manual Assignment Preparation
Pattern Type: relative N highest
Context: Bid Interval
Relevant event types: Delivery Bid
Pattern parameter: N = 5; value = Ranking
Cardinality: Single deferred
Find the five
highest bids
within the
bid interval
Taken from
the Fast
Flower
Delivery use
case

44
Why do we need policies - A simple example:
heavy trading scenario
 Given:
A stream of events of a single
topic, about the activity in the
stock market for a certain stock.
An event is produced every 10
minutes when there is trade in
the stock.
Each event consists of: quote
(current stock-quote), volume (an
accumulated volume of traded
events within these 10 minutes).
A selection specification:
“trigger an automatic trade
program if the volume exceeds
300,000 3 times within an hour;
pass as an argument the last
quote and the sum of the 3
volume values”.
Event-Id Time-Stamp Quote Volume
E1 9:00 33.23
E2 9:10 33.04 320,000
E3 9:20 33.11 280,000
E4 9:30 33.01 400,000
E5 9:40 32.90 315,000
E6 9:50 33.04 320,000
E7 10:00 33.20 303,000
E8 10:10 33.33 219,000
E9 10:20 33.11 301,000
E10 10:40 33.00 210,000
E11 10:50 32.78 400,000
E12 11:00 32.70 176,000
How many times the trade programming is triggered ;
Which arguments are used in each triggering?
Why defining patterns is not that
easy? Because we need to tune up
the semantics

45
Pattern policies
Evaluation policy—This determines when the matching sets are produced
Deferred
Cardinality policy—This determines how many matching sets are produced
within a single context partition
Repeated type policy—This determines what happens if the matching step
encounters multiple events of the same type
Every - Override – First – Last – with max/min value of…

46
Pattern policies – cont.
Consumption policy—This specifies what happens to a participant event
after it has been included in a matching set
Consume Reuse Bounded
reuse
Order policy—This specifies how temporal order is defined
By occurrence time – by detection time – by stream position – by attribute

47
Our entire culture is context sensitive
In the play “The Tea house of the August Moon”
one of the characters says: Pornography
question of geography
•This says that in different geographical
contexts people view things differently
•Furthermore, the syntax of the language (no
verbs) is typical to the way that the people of
Okinawa are talking
When hearing concert people are not talking,
eating, and keep their mobile phone on “silent”.

48
Context has three distinct roles (which may be combined)
Partition the incoming events
The events that relate to
each customer are
processed separately
Grouping events together
Different processing for
Different context
partitions
Determining the processing
Grouping together events
that happened in the
same hour at the same
location

49
Context Definition
A context is a named specification of conditions that groups event
instances so that they can be processed in a related way. It assigns
each event instance to one or more context partitions.
A context may have one or more context dimensions.
Temporal
Spatial
State Oriented
Segmentation Oriented

50
Context Types Examples
Spatial
State Oriented
Temporal
Context
“Every day between 08:00
and 10:00 AM”
“A week after borrowing a disk”
“A time window bounded by
TradingDayStart and
TradingDayEnd events”
“3 miles from the traffic
accident location”
“Within an authorized zone in
a manufactory”
“All Children 2-5 years old”
“All platinum customers”
“Airport security level is red”
“Weather is stormy”

51
Context Types
Fixed location
Entity distance location
Event distance location
Spatial
State Oriented
Fixed interval
Event interval
Sliding fixed interval
Sliding event interval
Temporal
Context

52
The present: State of the practice in event
processing – languages, implementation issues,
challenges in implementing event processing
applications

53
An Observation The Babylon Tower symbolizes the tendency
Of humanity to talk in multiple languages.
The Event Processing area is no different: most languages in the industry really follow
the hammer and nails syndrome – and extended existing approaches
• imperative script language
• SQL extensions
• Extension of inference rule language
The epts language analysis workgroup is aimed to understand the various styles
And extract common functions that can be used to define what is an event
processing language; this tutorial is an interim report
It does not seem that we’ll succeed to settle
In the near future around a single programming style

54
Existing Styles for EP languages (samples)
Inference
Rules
State
oriented
Agent
Oriented
Imperative/
Script Based
RuleCore
Oracle
Aleri Streambase
Esper
EventZero
TIBCO
WBE
Apama
Spade
AMiT
Netcool Impact
* - if we add simple and mediated event processing the picture is even more diversified
ECA
Rules
SQL
extension
Coral8
Agent Logic
Starview
XChangeEQ
Prova

55
StreamBase Studio

56
StreamBase Pattern Matching

57
CCL Studio (Coral8  Sybase)

58
CCL – Pattern Matching
RFID monitoring application
 Checks if a tag has been seen by readers A and B, then
C, but not D, within a 10 second window.
Insert into StreamAlerts
Select StreamA.id
From StreamA a, StreamB b, StreamC c, StreamD d
Matching [10 seconds: a && b, c, !d]
On a.id = b.id = c.id = d.id

59
Microsoft Streaminsights
var topfive = (from window in inputStream.Snapshot()
from e in window
orderby e.f ascending, e.i descending
select e).Take(5);
var avgCount = from v in inputStream
group v by v.i % 4 into eachGroup
from window in eachGroup.Snapshot()
select new { avgNumber = window.Avg(e => e.number) };

60
Esper EPL – FFD Example
/*
* Not delivered up after 10 mins (600 secs) of the request target
delivery time
*/
insert into AlertW(requestId, message, driver, timestamp)
select a.requestId, "not delivered", a.driver, current_timestamp()
from pattern[
every a=Assignment  (timer:interval(600 +
(a.deliveryTime-current_timestamp)/1000) and
not DeliveryConfirmation(requestId = a.requestId) and
not NoOneToReceiveMSG(requestId = a.requestId))
];

61
ruleCore - Reakt
 Event stream view - a unique context of events
 a view contains a window into the inbound stream of events and
contains commonly only semantically related events
 Situation - an interesting combination of multiple events as they occur over time
 An item with an RFID tag being picked up from the shelf and then moving past the checkout without being paid for
 Rule - an active event processing entity reacting to specific combinations of inbound events over time
 Action - the last part of a rule's evaluation in response to a detected situation

62
Amit Terminology
Situation
Conditions
Lifespan
e5 e8
Initiator Terminator
Event Selection Actions
Operation
Keys
e1
e3
Input
events
Operator
Joining
Counting
Temporal
Absence
Aggregation
Notifications
Messages
Definition updates
User plug-ins
e2

63
Amit - Situation

64
IBM Websphere Business Events

65
Apama EPL – FFD Examples

66
Apama Simulation Studio – cont.

67
Non functional properties - scalability
 scalability in the volume of processed events
 Scalability in the quantity of agents
 Scalability in the quantity of producers
 scalability in the quantity of consumers
 scalability in the quantity of context partitions
 Scalability in context state size
 Scalability in the complexity of computation
 scalability in the processor environment
Scalability is the capability of a system to adapt readily to a greater or lesser intensity of use, volume,
or demand while still meeting its business objectives.

68
Non functional requirements - security
 Ensuring only authorized parties are allowed to be event producers
of event consumers
 Ensuring that incoming events are filtered so that authorized
producers cannot introduce invalid events, or events that they are
not entitled to publish
 Ensuring that consumers only receive information to which they
are entitled. In some cases a consumer might be entitled to see
some of the attributes of an event but not others.
 Ensuring that unauthorized parties cannot add new EPAs to the
system, or make modifications to the EPN itself (in systems where
dynamic EPN modification is supported)
 Keeping auditable logs of events received and processed, or other
activities performed by the system.
 Ensuring that all databases and data communications links used by
the system are secure.

69
Performance objectives
Objective
Number
Objective Name Objective metrics
1 MAX input
throughput
Maximize the quantity of input events processed by a
certain system or sub-system within a given time period
2 MAX output
throughput
Maximize the quantity of derived events produced by a
certain system or sub-system within a given time period
3 MIN average
latency
Minimize the average time it takes to process an event
and all its consequences in a certain system or sub-
system
4 Min maximal
latency
Minimize the maximal time it takes to process an event
and all its consequences in a certain system or sub-
system
5 Latency
leveling
Minimize the variance of processing times for a single
event or a collection of events in a certain system or sub-
system
6 Real-time
constraints
Minimize the deviation in latency, from a given value,
for the processing of an event and all its consequences in
a certain system or sub-system.

70
Optimizations
Optimizations related to EPA assignment: partition, parallelism,
distribution and load shedding.
Optimization related to the execution process: scheduling, routing
optimizations and load shedding
Optimizations related to the coding of specific EPAs: code optimization,
state management.

71
Putting derived events in order

72
Ordering in a distributed environment -
possible issues
The occurrence time of an event is accurate, but the event
arrives out-of-order and processing that should have
included the event might already been executed.
Neither the occurrence time nor detection time can be trusted,
so the order of events cannot be accurately determined.

73
Solutions:
Buffering techniques:
 Assumptions:
 Events are reported by the producers
as soon as they occur;
 The delay in reporting events to the
system is relatively small, and can
be bounded by a time-out offset;
 Events arriving after this time-out
can be ignored.
 Principles:
 Let  be the time-out offset,
according to the assumption it is safe
to assume that at any time-point t,
all events whose occurrence time is
earlier than t -  have already
arrived.
 Each event whose occurrence time
is To is then kept in the buffer until
To+, at which time the buffer can be
sorted by occurrence time, and then
events can be processed in this
sorted order.
Retrospective
compensation:
 Find out all EPAs that have
already sent derived events
which would have been
affected by the "out-of-
order" event if it had arrived
at the right time.
 Retract all the derived
events that should not have
been emitted in their current
form.
 Replay the original events
with the late one inserted in
its correct place in the
sequence so that the correct
derived events are
generated.

74
Inexact event processing
 uncertainty whether an event actually occurred
 inexact content in the event payload
 inexact matching between derived events and the situations they purport to
describe
Source
malfunction
Malicious
source
Projection of
temporal
anomalies
Imprecise
source
Sampling or
estimate
Propagation
of
inexactness
Uncertain
event
Inexact event
content

75
False positives and false negatives
False positive situation detection refers to cases in which an
event representing a situation was emitted by an event
processing system, but the situation did not occur in reality.
False negative situation detection refers to cases in which a
situation occurred in reality, but the event representing this
situation was not emitted by an event processing system

76
Retraction
Delivery
Request
issued
Bid
Request
issued
Assignment
issued
Pick-up
occurred
Delivery
occurred
Order Cancellation
Cancel
bid
preparation
Cancel
bid
and
assignment
process
Notify
assigned
driver
about
cancellation
Undoable
May
still
be
cancelled
with
some
penalty

77
The future: Trends and research challenges

78
Trend I: Going from narrow to wide
Border security radiation detection
Mobile asset geofence
Logistic and scheduling
Unauthorized use of heavy
machinery
Hospital patient and asset tracking
Activity monitoring for taxing and
fraud detection
Intelligent CRM in banking
EDA and asynchronous BPM in
retail
Situation awareness in energy
utilities
Situation awareness in airlines
Reduce cost in injection therapy
Next generation navigation
Real-time management of
hazardous materials
Finding anomalies in point of sales
in retail stores
Elderly behavior monitoring
Some recently
reported applications
(EPTS use-cases WG)
Source: ebizQ Event processing market pulse

79
Trend I: Going from narrow to wide
Taking event processing outside enterprise computing:
Robotics
Bio-Informatics
Socio-technical systems
Home
Automation

80
Trend II: Going from monolithic to
diversified
Variety of functions
Variety of Quality of Service
requirements:
Variety of platforms
“One size fits all” will not work –
Instead a collection of building blocks
that can fit together

81
Trend III: Going from proprietary to
standard-based – standard directions
The current situation: Babylon tower: variety of
languages, event representation…
Serves as enabler to achieving other trends and
general maturity
The shift of vendors from start-up dominant to
bigger companies makes the atmosphere more
friendly towards standards.
Areas for
Standards:
Modeling
Event representation
Interoperability
Languages PIM

82
Trend IV: Going from programmer centered to
semi-technical person centered
Source: ebizQ Event processing market pulse

83
Trend V: Going from stand-alone to
embedded
Packaged applications
Business Activity
Monitoring
Sensor Platform
Middleware and platforms

84
Trend VI: Going from reactive to proactive
TRAFFIC JAM
End of game

85
Emerging directions: Four directions to observe
Multiple platforms – same
look and feel
The engineering of
constructing EP
applications
Adding intelligence to
Event processing
Tailor-made optimizations

86
Emerging direction I: multiple platforms – same
look and feel
Appliance
Appliance
Stream
Platform
Stream
Platform
Cloud
computing
Platform
Cloud
computing
Platform
ESB /
Messaging
Platform
ESB /
Messaging
Platform
Embedded
Embedded
Virtual Event Processing Platform

87
Emerging direction II: Tailor-made
optimizations
 Local optimizations: each EPA will be optimized for its own
purpose / assumptions / QoS indicators – average/worst
case latency, input/output throughput …
 Global optimization: scheduling, load balancing,
assignment…
EPA EPA
EPA
EPA
Producer
Producer
EPA
Consumer
Consumer
Local optimizations
Global optimizations

88
Emerging direction III:
Event processing software engineering
Best practices
Methodologies
Modeling
& meta-modeling
Design
Patterns

89
Emerging direction IV:
Intelligent event processing
Offline and
continuous mining
of meaningful
patterns in event
histories
Inexact event
processing –
handling inexact
events and also
false positives and
false negatives
Causality – a key
for proactive, but
also vital for
provenance

90
Summary
Event processing
has emerged from
some academic
disciplines
The state of the
Practice is the 1st
generation of
products – mainly
engineering based
Going to the next
phase – many
challenges that
require collaboration
of research and
industry
Already attracted coverage of
analysts and all major software
vendors

OpherEtzion Wed-PM.ppt

More Related Content

Similar to OpherEtzion Wed-PM.ppt

Recently uploaded

OpherEtzion Wed-PM.ppt