Part 1 A Simple Introduction to Complex Event Processing

The Power of Event
Part 1
A Simple Introduction to
Complex Event Processing
Not For Me(@JoeWooJin)
jwj0831@gmail.com

The Power of Event
Chapter 1
The Global Information Society and
the Need for New Technology
jwj0831@gmail.com

Contents
● Events everywhere in our information systems
● The Internet and the growth of global communication
spaghetti
● Layer upon layers in enterprise system architecture
● Global electronic trade - understanding what is
happening
● Agile systems - future reality or just a dream
● Can an open electronic society defend itself?
● The gathering storm - global coordination or global
chaos

1.1 Distributed Information Systems
Everywhere
● The Internet has promoted and speeded the growth of
distributed information processing beyond the single
enterprise, across the boundaries between enterprises.
● Financial Trading System, government and military
information system
● underlying architectural strucuture is the same: a
distributed information system, called "enterprise
systems"
● business-level, or stragetic-level events; complex
events.

1.2 The Global Communication
Spaghetti Pot
● We can't tell where it starts and where it ends, we can't
unravel it, and most of the time we don't know how it
happend.
● The challenge is not to restrict communication flexibility,
but to develop new technologies to understand it
● The technology of monitoring and managing events in
IT systems has been completely overrun by the
technology of communicating events.
● This is a global problem in search of new ideas on how
to solve it.

1.2.1 Event Causality
● A key to understanding events is knowing what caused
them.
● Events are flowing all the time through our Internet-
based systems from one part of the world and causing
events in another part of the world.
● Horizontal causality
● Vertical causality

1.3 Electronic Archeology: Layers
upon Layers
● Enterprise systems are distributed, event-driven
systems.
● They are layered systems.
● Layering is a design technique for controlling
complexity.
● Layered IT systems present another dimension in the
search for new ways to understand the events that
happen in them.

1.3.1 A Layered Enterprise System
● Corporate Level
● Applications and User at the Top; business level or
strategic planning level
● Collaboration and Enabling Layer
● Middleware
● Network Layer

1.3.2 Vertical Causality: Tracking
Events up and down the Layers
● Acitivity at each layer is translated into activities at the
layers below and conversely.
● An activitiy at the top causes activities at successively
lower levels, which in turn cause other activities to
happen at the top.
● An ability to track vertical causality in a layered system
a. understanding properties of that high-level event,
such as its timing and how it improve performance at
the business level.
b. to group events at lower levels according to the high-
level events they signify.

1.3.3 Event Aggregation: Making High-Level
sense out of Low Level Events
● The complementary operation to downward tracking of
vertical causality is the aggregation of sets or groups of
lower events into a single higher-level event that
expresses the meaning of the lower-level events, taken
together.
● Recognizing or detecting a significant group of lower-
level events from among all the enterprise event traffic,
and creating a single event that summarizes in its data
their significance, is called event aggregation.

1.4 The Gathering Storm of New
Activities on the Web
● Widely distributed fragments of information must be
gathered and pieced together into forms appropriate for
elctronic processes to tackle the problems at hand, and
for humans to understand high-speed business
situations and stay in control.
● We must be able to deal with increasingly complex
systems, demand for high reliablity, and frequent, rapid
modifications.
a. Global Electronic Trade
b. Agile Systems
c. Cyber Warfare and the Open Electronic Society

The Power of Event
Chapter 2
Managing the Electronic Enterprise
in the Global Event Cloud
jwj0831@gmail.com

Contents
● The global event cloud
● How enterprises operate in the global event cloud
● Management process - going beyond workflow
● Autonomous parallel, asynchronous process
● The electronic enterprise
● Treating the exceptional situation as normal
● Enabling the human to control the electronic enterprise
● Technology demands for managing the electronic
enterprise

2.1 How the Global Event Cloud
Forms
● The Open Enterprise
○ Their boundaries become blurred.
○ Events are flowing between the enterprise and its
trading partners, the electronic market hubs...
● The Global Event Cloud
○ a "cloud" of events rather than a "stream" because
the event traffic is not, in most cases, nicely
organized.
● The Electronic Enterprise
○ "Event driven" - those tools and applications rely on
receiving events to monitor he progress of a process
and issuing eventss to initiate its next stage.

2.2 Operating in the Global Event
Cloud 1/2
● Enterprise management processes
today typically consist of a large
number of linear workflows like this
one, loosely strung together and
nested one inside another
● ProcessOrder is a process for handling
incoming orders from customers in our
typical elctronic enterprise.
● It is an electronic middleman, perhaps
using eMarketplaces to select its
vendors.
ProcessOrder
1. New Order
2. Select Vendor
3. Pick, Pack,
and Ship Product
4. Billing

Spare Part
Query
Terminate
Instance
Select Product
Ship Reqeust
GL Query
Invoice Print
Ship Quote
Ship Order
Invoice Print
Serice
Requeset
Valid ServiceProcess Order
Cloud 2/3
1. New Order
2. Select Vendor
3. Pick, Pack,
and Ship Product
4. Billing
Process Activities
Valid Order When a valid order is
received, select the
vendor with the lowest
price.
When a vendor is
selected, send a ship
order to the vendor's
warehouse.
When a UPS routing
number is received,
terminates the
process.
Process RulesGlobal Event Cloud
Select Vendor
Vendor
Selected
Ship Order
Product
Shipped
Terminate
Instance

Cloud 3/3
● In Figure the ProcessOrder workflow is on the left, the
global event cloud in the center, and the rules that drive
the workflow process are on the right.
● The rules that drive the process are "reactive rules".
● Each rule contains an event pattern called a trigger.
● These events that drive the workflow process are
transported back and forth between the activities and
the workflow rules engine by the enterprise IT layer.
● They are part of event cloud.

2.3 Going Beyond Workflow
● To enter the world of global Internet marketplace
decision making, business and management process
must meet the reduced time scales and increased
situational complexities that will be involved.
● They soon will be parallel, asynchronous processes.
● "Autonomous parallel processing"
○ Be completely automated and event driven.
○ Execute in parallel.
○ Make decision and communicate asynchronously.

2.4 Parallel and Asynchronous
Process
● Scalable complex event pattern matching, taking into
account the context or state at the time of a match
● The ability to reuse the data in sets of events that match
a rules pattern in the creating of new events.
● These technology demands are the foundation for
implementing parallel, asynchronous real-time
management processes.

2.5 On-the-fly Process Evolution
● On-the-fly evolution means the ability to modify a
process without halting the rules engines or disrupting
the execution of other processes
● Technology demands for controlling and modifying
electronic processes are
○ Real-time, personalized viewing of activity at every
level in the enterprise.
○ On-the-fly process modification
○ Simulation of processes before going live

2.6 Exceptions Must Be First-Class
Citizens in Process Design
● If a process fails to behave in a given situation as
specified or meets a situation for which it has no
specification, it is said to have encountered an
exception.
● So, the process design technology should let us design
normal processing and exceptional processing in the
same way.
● However, there are some distinguishing problems in
dealing with exceptions
○ We must be made aware of their presence in real
time.
○ We must be able to find out what causes them

The Power of Event
Chapter 3
Viewing the Electronic Enterprise -
Keeping the Human in Control
jwj0831@gmail.com

Contents
● Event monitoring - the standard technology today
● Enterprise viewing - a step beyond monitoring
● Recognizing sets of events from the global event cloud -
a key to personalized viewing
● Information gaps
● Enterprise structure and abstraction hierarchies
● Hierarchical viewing - the key to human control of the
enterprise

3.1 Today's Event Monitoring Is Too
Primitive
● System Monitoring Focuses on the network layer.
● Network-level monitoring doesn't even solve network
problems.
● The tools contains very littel "smarts" to tell a manager
what the problem is and what to do.
● They are faced daily with the following kinds of issues.
○ The network event logs can become very large and
difficult to handle in real time.
○ Tools to aid in picking out sets of related events are
needed.
○ Causal tracking is needed.
○ Predictive monitoring is beyond the state of the art.

3.3 Information Gaps
● Different people engaged in the operations of an enterprise need
different kinds of information.
● This leads to information gaps between the kind of information
people need to do their jobs effectively and easily, and the
information they actually get.
● An information gap generally has two dimensions:
○ A vertical dimension, which is the difference between the level
of the enterprise at which events and other data are monitored
and the level at which the user is operating within the
enterprise
○ A horizontal dimension, which is the amount of analysis needed
to render the monitored information in a userful form for the
user's tasks/

3.4 Problem-Relevant Information
● To bridge information gaps we need a technology for
constructing problem-relevant information from
whatever events we can monitor.
● How to get problem-relevant information
1. Relevance tothe problem of immediate interest
2. Ease of understandability
3. Ease of analysis
4. Ease with which multiple views can be coordinated

3.5 Viewing Enterprise Systems 1/2
● A view of a system is a selection of information about
what the system is doing currently or did in the past that
is processed to abstract or extract those aspects
relevant to a problem of interest.
● Examples
○ An application-level activity view
○ A who's-talking-to-whom view
○ A network problems view
○ A high-level system performance view
○ A corporate business process tracking view

3.5 Viewing Enterprise Systems 2/2
● Each of these examples of a view has the following
elements
○ Each view has a problem of interest.
○ Each view is event driven.
○ The view are provided in humanly understandable
forms using graphics
○ Each view provides relevant events that can be used
to drive automated decision making processes.
○ Most important, a view must be easy to modify, on
the fly, to incorporate new types of events, change
the aggregation technique

3.6 Creating and Coordinating
Multiple Views
● Different people need different views.
● Simply, this is because different users are interested in
different kinds of information about the system.
● Not only do we need multiple views of a system, but
each user needs to be able to customize their own view.

3.7 Hierarchical Viewing
● A powerful technique to help in understanding a
complex enterprise system is to seperate the system's
activities, and the operations that implement those
activities, into layers-called levels. This is called an
abstraction hierarchy.
● Viewing a system's behavior at different level is called
hierarhchical viewing.
● To build hierarchical views we must first define an
abstracttion hierarchy.
○ Operational description
○ Hierarchical structuring

The Power of Event
Chapter 4
Designing the Electronic Enterprise
jwj0831@gmail.com

Contents
● Rapid process modification to meet the demands of the
eMarketplace
● The roles of process architecture in the process lifecycle
● Using process architectures to reduce errors in mission-
critical process systems
● Constituents of process architecture-reactive,
behaviors, design constraints
● Dynamic process architectures
● Layered architectures and plug-and-play systems
● The abstraction principle for layered architectures

The Power of Event
Chapter 5
Events, Timing, and Causality
jwj0831@gmail.com

Contents
● What events are
● How events are created
● The form, signinficance, and relativity of an event
● Timing, causality, and aggregation
● Genetic parameters of events
● Partial orderings of events
● Timing requirements expressed as patterns of events
● Examples of causal tracking of interprocess activity

5.1 What Events are
● An event is an object that is a record of an activity in a
system. An event may be reated to other events.
● An events has three aspects
○ Form: The form of an event is an object.
○ Significance: An event signifies an acitivity.
○ Relativity: An activity is related to other activities by
time, causality, and aggregation.

5.2 How Events Are Created
● Two Steps to create events
1. Observation step
2. Adaptation step
● Three prinicipal sources of events
1. IT layer
2. Instrumentation
3. CEP

5.3 Time, Causality, and Aggregation
● Time: Time is a relationship that orders events
● Cause: If the activity signified by event, A had to
happen in order for the activity signified by event B to
happen, then A caused B. Causality is a dependence
relationship between activities in a system.
● Aggregation: If event A signfies an activity that consist
of the activities of a set of events, B1, B2, B3, ..., then A
is an aggregation of all the events Bi. Conversely, we
say events A and B are indepenent if neither caused the
other.

5.3.1 The Cause-Time Axiom
● All these relationship between events have some simple
mathematical properties. They are transitive and
asymmetric
● Cause-time axiom: If event A caused event B in system
S, then no clock in S gives B an earlier timestamp than
it gives A.

5.4 Genetic Parameter in Events
● Special data parameters are added to an event to
encode its timing and its casual relationships to other
events. They are often referred to as genetic parameter
● Timestamps
● Casual vector: A parameter of an event that contains
the identifiers of the set of events that caused the event.

5.5 Time 1/2
● A timing relationship between events created by a
system tells how the system is performing.
● Timing is also an important filter in debugging
● By using CEP, we can easily deal with the issue that
their order of observation might be different from their
order of execution.
● The concept that makes this possible is event pattern
matching.
● Event patterns are a powerful way to state our
deadline requirement simply and concisely.

5.5 Time 2/2
● CEP rules like this example are called constraints
because their purpose is to check a system's
performance for violation of requirements
Rule: Check Timely Delibery
Element Declarations
Variable Subject S, Message M, String Id, Time T, T1, T2
Event types Publish(Subject S, String Id, Message M, Time T),
Receive(Subject S, String Id, Message M, TIme T),
Warning(String Id, Time T)
Relational operators and
Pattern Publish(StockTrade, Id, M, T1) and Receive(StockTrade, Id, M, T2)
Context test T2 - T1 >= 10 mins
Action create Warning(Id, T2 - T1)

5.6 Causality and Posets
● Events in a distributed system happen in a relationshp
of dependence or independence. This relationship,
called causality, plays an important role in any kind of
analysis of what happened in a system, either online or
postmortem.
● A set of events together with their causal relationship is
called poset, partially ordered set of events
● The essential role of causality is to focus the search
space for the causes of some activity.

5.7 Casual Event Execution-Real-
Time Posets
● A causal event execution is a poset consisting of events
generated by a system and their relationships.
● We use "causal event execution" tp emphasize that we
are dealing with events and their relativities online real
time.

5.8 Orderly Observation
● If their time of arrival is consistent with their causal
relationship in the target system, we say that the
observation is orderly.
● where A --> B means "A causes B", and the ArrivalTime
of an event is its time of arrival at a CEP adapter,
assuming adapters have synchronized clocks.
Orderly observation: for any pair of events A and B,
if A --> B then ArrivalTime(A) <= ArrivalTime(B)

5.9 Observation and Uncertainty 1/2
● The events created by CEP can be thought of as event
inferences drawn from observed events.
● CEP may also process these event inferences and use
them to create more inferences.
● Event inferences can signify activities within the system
that were not signified by events observed in the system
itself.

5.9 Observation and Uncertainty 2/2
● The totality of what we can observe, together with what
we can infer, is called observable system.
○ Observable System: The set of all events that can
be observed from a target ssytem, or inferred from
observations, is called the observable system.
○ Uncetainty Principle
■ An activity within a target system may not have
any signifying event in the observable system.
■ We must always bear in mind that there may be
activities going on in the system that we cannot
know about.

The Power of Event
Chapter 6
Event Patterns, Rules, and
Constraints
jwj0831@gmail.com

Contents
● Familiar kinds of pattern searching
● Event patterns
● A strawman event pattern language
● Event pattern rules
● Event pattern constraints
● Capturing business rules as event patterns

6.1 Common Kinds of Pattern
Searching
● We need to be able to describe a pattern of events that
we are interested in and quickly find the sets of events
that match the pattern.
● To do this, we first need a precise method to describe
event patterns.
● One way is to write the pattern in a computer language
called an event pattern language(EPL).
● Another way, which many of us are familiar with, is to
use a graphical user interface(GUI) such as those
provided in popular Web search engines.
● Unfortunately, search GUIs usually allow only very
simple patterns to be described

6.2 Event Patterns 1/2
● An event pattern is a template that matches certain sets
of events - the sets you want to find.
● It describes precisely not only the events but also their
causal dependencies, timing, data parameters, and
context.
● So an event pattern is a template for posets.

6.2 Event Patterns 2/2
● Some examples of event patterns are the following:
a. All orders from custom C in the last month (content-sensitive)
b. All orders from frequent customers in the last month (context-
sensitive)
c. All orders from customers in response to a discount
announcement (filter)
d. All orders from customers at the regular price that have led to
the customer requesting a reduced price in response to the
discount announcement (complex)

6.3 A Strawman Pattern Language
● A pattern in STRAW-EPL must declare the following
four elements:
○ A list of variables.
○ A list of types of events.
○ A pattern.
○ A condition on the context of any match

6.4 Event Pattern Rules 1/2
● An event pattren rule is reactive rule that specifies an
action to be taken whenever an event pattern is
matched.
● An event pattern rule implies a causal relationship
between the events that trigger it by matching its pattern
and the events that are created when the rule executes
its action.
● A reactive rule has two parts:
○ A trigger, which is an event pattern
○ An action, which is an event that is created
whenever the trigger matches

6.4 Event Pattern Rules 2/2
● Reactive rules can be either sequential or parallel.
○ A sequential rule implies that all its triggerings take
place in a sequence, one after the other.
○ A parallel rule implies that its triggerings take place
independently, as if executed by new threads of
control.
○ A sequential rule uses create to indicate its action,
whereas a parallel rule uses create parallel.

6.5 Constraints 1/2
● Constraints can be used to specify not only how a target
system should behave, but also how its user should use
it.
● A never constraint consists of the following:
○ The temporal operator never
○ A STRAW-EPL pattern

6.5 Constraints 2/2
● The purpose of a rule is to create new events in
response to situations.
● The purpose of a constraints is different, simply to
monitor for a situation.
● Typically, a constraint is used to express a requirement
on system behavior that is not guaranteed by the
system.

The Power of Event
Chapter 7
Complex Events and
Event Hierarchies
jwj0831@gmail.com

Contents
● Event aggregation
● Complex events
● Event abstraction hierarchies
● Personalized and role-baed viewsof hierarchical
systems

7.1 Aggregation and Complex
Events
● A complex event is an aggregation of other events,
called its members as defined in Section 5.3.
● The relationship between a complex event and its
members is called aggregation.
● A complex event can signify an acitivity that consists of
several activities in different parts of a distributed
system.
● Conceptually, we think of a complex event as an event
at a higher level thatn the levels of its members.

7.2 Creating Complex Events 1/3
● Event pattern rules are used in CEP to create complex
events siginifying the acitivities of sets of events.
● This user of event pattern rules gives us a powerful
method of recognizing significant high-level events from
among a cloud of low-level events.
● Aggregation is a tool for making the activities in a
complex system understandable to humans.

● One of the reasons we need event patterns to create complex
events is that a complex event can happen in more than one way.
● We need more powerful pattern lanuguage than STRAW-EPL.
● The lower-level events that are aggregated by a complex event can
be thought of as causing the complex event.
● Our primary use of complex events is to view the activities in the
system as happening at a series of abstraction levels.
● Hence we distinguish between causality, which is a relationship
between events at the same level of abstraction, and aggregation,
which is an abstraction relationship.
● We sometimes refer to aggregation later on as vertical causality.

● Example
Rule: Completed Data Transfer
Element Declarations
Variable Node N1, N2, Data D, Bit B, Time T1, T2, T3, T4
Event types Send(Node N1, Node N2, Data D, Bit B, Time T1),
Receive(Node N1, Node N2, Data D, Bit B, Time T1),
Ack(Node N1, Node N2, Bit B, Time T1),
RecAck(Node N1, Node N2, Bit B, Time T1),
CompletedTrans(Node N1, Node N2, Data D, Time T1, Time T2)
Relational operators -> (causes)
Pattern Send(N1, N2, D, B, T1) -> Receive(N2, N1, D, B, T2) -> Ack(N2, N1, B,
T3) -> RecAck(N1, N2, B, T4)
Context test T4 - T1 <= TimeBound
Action create CompletedTrans(N1, N2, D, T1, T4)

7.3 Event Abstraction Hierarchies
● In CEP an event abstraction hiearchy consists of the following
elements.
a. A sequence of levels of activities: Each level consists of a set
of descriptioins of system acitivities and, for each activities, a
specification of the types of events that signify instances of that
acitivity. Level 1 is the lowest leve
b. A set of event aggregation rules for each level: For each level
(except level 1), there must be a rule for creating each type of
event at that level as an aggregation of events at levels below.
● The crucial aspect of this definition is the set of rules specifying
how each event at a higher level is an aggregation of events at
levels below it.

7.4 Building Personalized Concept
Abstraction Hierarchies
● We use event abstraction hierarchies to specify and implement
personalized views of a target system for each stakeholder in the
system.
● This is a two step process following the definition in Section 7.3.
● Aggregation rules are the key to the second step.
a. The types of events that we choose to view at each hierarchical
level must be specified as types of CEP events. This is the step
where "personalization" of views takes places.
b. An aggregation rule must be defined for each type of event at
each level above level 1.

Part 1 A Simple Introduction to Complex Event Processing

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (20)

Similar to Part 1 A Simple Introduction to Complex Event Processing

Similar to Part 1 A Simple Introduction to Complex Event Processing (20)

Recently uploaded

Recently uploaded (20)

Part 1 A Simple Introduction to Complex Event Processing