Academic Paper Writing Service http://StudyHub.vip/10201-Executive-Summary-And-Manifesto-E 👈

1. 1
The event processing m anifesto
W ritten by the participants of
the 2 0 1 0 Dagstuhl sem inar on
event processing
http: / / www.dagstuhl.de/ 10201
Dagstuhl Seminar Proceedings 10201
Event Processing
http: / / drops.dagstuhl.de/ opus/ volltexte/ 2011/ 2985

2. 2
TABLE OF CONTENTS
PREFACE........................................................................................................ 5
EXECUTIVE SUMMARY ..................................................................................... 6
Chapter 1: Why event processing? .................................................................... 7
1.1 What is event processing?........................................................................ 7
1.2 Who should read this? ............................................................................. 7
1.3 Origins of event processing ...................................................................... 8
1.4 Problem s solved by event processing......................................................... 8
1.4.1 Manufacturing Execution system s ........................................................ 9
1.4.2 Location-based services ................................................................... 10
1.4.3 Algorithm ic trading .......................................................................... 10
1.4.4 Defense intelligence......................................................................... 11
1.5 Criteria for adopting an event processing approach.................................... 12
1.6 Benefits............................................................................................... 15
1.6.1 Event processing system s provide inform ation faster............................ 15
1.6.2 Event processing system s im prove the quality of available inform ation ... 15
1.7 Benefits of event processing system s ...................................................... 16
1.7.1 Operational changes can be m ade sooner, m aking them m ore effective .. 16
1.7.2 Radically new, tim e-sensitive business practices m ade possible ............. 17
1.7.3 Tim ely inform ation dissem ination leads to better decisions.................... 18
1.7.4 Staff cost is reduced by offloading work to com puters .......................... 19
1.8 Build-versus-buy .................................................................................. 19
1.9 Sum m ary – the value of event processing ................................................ 21
Chapter 2: What are the characteristics of event processing? ......................... 22
2.1 What do we m ean by event processing?................................................... 22
2.2 Characteristics of event processing applications ........................................ 23
2.3 Event processing application requirem ents ............................................... 23
2.3.1 Event I nput/ Output ......................................................................... 24
2.3.2 Data Reduction ........................................................................... 24
2.3.3 Reasoning: ..................................................................................... 24
2.3.4 Context awareness ......................................................................... 24
2.3.5 Logging and analysis ....................................................................... 25
2.3.6 Prediction ....................................................................................... 25
2.3.7 Learning and Adaptation ............................................................... 26
2.3.8 Distribution .................................................................................... 26
2.4 Non-functional requirem ents .................................................................. 26
2.4.1 Perform ance ................................................................................... 26
2.4.2 Availability and Recoverability ........................................................... 27
2.4.3 Consistency and I ntegrity in a distributed system ................................ 28
2.4.4 Security and privacy ........................................................................ 28
2.4.5 Usability/ Maintainability/ Manageability ........................................... 29
2.5 Modeling event processing applications ............................................... 29
Chapter 3: Synergies and relations to other areas ............................................. 31
3.1 An event-driven theft detection as an exam ple of synergies and relations to
other areas ............................................................................................... 31
3.2 Synergies and relations of event processing and analytics .......................... 31
3.2.1 Learn and discover event processing patterns with predictive analytics ... 32
3.2.2 I m plem enting and executing predictive m odels with event processing .... 34
3.2.3 I ncorporate predictive m odels in event processing ............................... 34
3.3 Synergies and relations of event processing and active rules (ECA) ............. 34
3.4 Synergies and relations of event processing and publish-subscribe approach. 35
3.4.1 Integrating publish-subscribe with event processing ......................... 36

3. 3
3.5 Synergies and relations of event processing and business process m anagem ent
(BPM) ....................................................................................................... 37
3.5.1 I m plem enting BPM with event processing ........................................... 37
3.5.2 Monitoring processes with event processing ........................................ 37
3.5.3 I nfluencing business processes with event processing .......................... 37
3.6 Synergies and relationships of event processing and data stream s............... 38
3.6.1 Brief overview of stream s ................................................................. 38
3.6.2 Relationship of Stream s to Event Processing ....................................... 39
3.7 Event processing and business rules m anagem ent system s (BRMS) ............. 40
3.8 Sum m ary ....................................................................................... 40
Chapter 4: Event processing related standards.................................................. 41
4.1 The EP standards reference m odel .......................................................... 41
4.1.1 Business and technical perspectives................................................... 41
4.1.2 Dom ain-specific and general standards........................................... 42
4.2 Standards per the ESRM classification...................................................... 43
4.2.1 Com m on standards and laws......................................................... 43
4.2.2 Dom ain reference m odel............................................................... 43
4.2.3 Dom ain use case ......................................................................... 43
4.2.4 Strategy ..................................................................................... 43
4.2.5 Functional m odel .......................................................................... 43
4.2.6 Com puter-independent m odel ....................................................... 43
4.3 Platform independent m odel standards (PI M) ............................................ 44
4.4 Standards in ESRM areas: ...................................................................... 45
4.5 Next steps (action item s) ....................................................................... 47
Chapter 5: Grand challenge: The global event processing fabric and its applications
.................................................................................................................. 48
5.1 The Event Processing Fabric grand challenge ............................................ 48
5.2 Event Processing Fabric im plem entation issues ......................................... 49
5.2.1 The Fabric .................................................................................. 49
5.2.2 Quality attributes......................................................................... 50
5.2.3 Business and societal adaptation ................................................... 50
5.3 Applications utilizing the Event Processing Fabric ...................................... 51
5.4 Elem ents of the challenge ...................................................................... 51
5.5 Related work ........................................................................................ 51
5.6. Sum m ary ........................................................................................... 52
Chapter 6: Near-term research ....................................................................... 53
6.1 Event sem antics ................................................................................... 53
6.1.1 Probabilistic events.......................................................................... 53
6.1.2 Provenance .................................................................................... 54
6.1.3 Event context ................................................................................. 54
6.2 Events and actions................................................................................ 54
6.2.1 Com plex actions.............................................................................. 55
6.2.2 Goal-directed reaction...................................................................... 55
6.2.3 Com pensation and retraction ............................................................ 55
6.2.4 Predictions and speculations ............................................................. 56
6.2.5 Adaptive event processing ................................................................ 56
6.3 Event processing system s ...................................................................... 56
6.3.1 Function placem ent and optim ization ................................................. 56
6.3.2 Consistency .................................................................................... 57
6.4 Privacy and security .............................................................................. 57
6.4.1 Access control................................................................................. 57
6.4.2 Authenticity of events ...................................................................... 58
6.4.3 Privacy .......................................................................................... 58

4. 4
6.3 Sum m ary ............................................................................................ 58
REFERENCES ............................................................................................. 59

5. 5
PREFACE
The second Dagstuhl sem inar on event processing took place in May 2010. This five-
day m eeting was oriented to work toward a com prehensive docum ent that would
explain event processing and how it relates to other technologies and suggest future
work in term s of standards, challenges, and shorter-term research projects.
The 45 participants cam e from academ ia and industry, som e of them out of the
event processing field. The team s continued the work after the conference and have
sum m arized their findings in this docum ent. The chapters were written by different
team s and then edited for consistency.
The chapter team leaders were: Robert Berry (Chapter 1), Peter Niblett (Chapter 2),
Arno Jacobsen (Chapter 3), Paul Vincent (Chapter 4), Bernhard Seeger (Chapter 5),
and Patrick Eugster (Chapter 6). The Dagstuhl sem inar was organized by Rainer von
Am m on, Mani Chandy, and Opher Etzion (who served as editor for this docum ent);
the technical editing was done by Sharon Geva.
The following people participated in the sem inar:
Rainer von Am m on, Darko Anicic, Stefan Appel, Jean Bacon, Robert Berry, Pedro
Bizarro, Andrey Brito, Sim on Brodt, Francois Bry, Alejandro Buchm ann, Sharm a
Chakravarthy, Badrish Chandram ouli, Mani Chandy, Christoph Em m sersberger,
Opher Etzion, Patrick Eugster, Dieter Gawlick, Annika Hinze, Martin Hirzel, Mark
Horsburgh, Arno Jackobsen, Boris Koldehofe, Alexander Kozlenkov, Wolfgang May,
Daniel Meiron, Ken Moody, Peter Niblett, Adrian Paschke, Udo Pletat, Olga Poppe,
Tore Risch, Harold Shcoening, Roy Schulte, Bernhard Seeger, Marco Seirio, Guy
Sharon, Plam en Siem onov, Florian Springer, Nenad Stojanovic, John Sutcliffe-
Braithwate, Richard Tibbetts, Ronen Vaisnberg, Paul Vincent, Agnes Voisard,
Christian Wolff, Carlo Zaniolo, and Holger Ziekow.
All participants contributed to this docum ent.
This role of this docum ent is twofold:
To educate the public about event processing, since it is a relatively new area
To call for action to the com m unity in the areas of standards and further research
One of the highlights of the sem inar was the establishm ent of an event processing
grand challenge. We believe that the current applications based on event processing
technology just scratched the surface of its potential; the grand challenges offers a
focus to m ake the quantum leap in the im pact of event processing on the world.

6. 6
EXECUTI VE SUMMARY
Event processing (EP) is an area in the field of inform ation technology that is central
to m any system s on which our society depends. These system s include energy,
healthcare, the environm ent, transportation, finance, services, and m anufacturing.
Event processing consists of m ethods and tools to filter, transform , and detect
patterns in events, in order to react to changing conditions, typically under som e
tim e constraints.
We present this docum ent to introduce the area of event processing, explain its
pertinence to other fields, and to provide inform ation to enable relevant business
opportunities. We also aim to establish guidelines for how event processing can fit
into current standards and to put forth short- and long-term goals for event
processing professionals in industry and academ ia.
Event processing system s perform the following four m ain functions:
Obtain data from m ultiple sources in real or near-real tim e
Aggregate and analyze this data to detect patterns that indicate the presence of
critical situations requiring a response
Determ ine the best response for such situations
Monitor the execution of that response
Why is event processing of increased im portance now, when even the earliest rule
engines and business processes had m echanism s used to detect critical situations
and respond accordingly?
Today's world is m uch m ore dependent on IT system s than it ever was. All of us are
m uch m ore interconnected and interdependent than ever. System s m ust be able to
react to events anywhere on the globe. An outbreak of Ebola on one continent, for
exam ple, dem ands a response in countries everywhere. Responses m ust occur ever
quicker, som etim es in m illiseconds, as the pace of the stock exchange illustrates.
The costs of inappropriate responses can be staggering, as we see in the cases of
certain defense applications. In m any telecom m unication system s, the volum e of
data that m ust be analyzed in near-real tim e is torrential. In addition, the variety
and types of data that m ust be analyzed in event processing system s is enorm ous.
Such data m ay be in the form of structured text, natural language, im ages, audio, or
video. The data m ay be delivered to the system , or it m ay have to be extracted by
the system . In m any system s, security is an overarching concern.
Com ing decades will see m any m ore applications with event processing capabilities,
as society dem ands sm arter ways for m anaging electric power, water, health, retail
and distribution, traffic, and safety—sm arter m eaning responding better and faster
to changing conditions. The interconnected nature of the m odern world m eans that
researchers, designers, and students can no longer develop event processing for a
single dom ain, such as the sm art grid, without incorporating developm ents related to
event processing technologies in other dom ains, such as sm art healthcare. To step
up to these challenges, we are in urgent need of event processing theory, design
m ethods, and tools. This docum ent is an im portant step toward that goal.

7. 7
Chapter 1 : W hy event processing?
In this chapter, we provide an introduction into event processing and describe our
m otivation for pursuing research in this field. We also provide insight into the value
of using an event-driven approach in various system s.
1 .1 W hat is event processing?
Today’s inform ation society abounds in m yriad inform ation flows, com puter-based
hum an collaborations, software agent interactions, electronic businesses, and the
explosion of data on the Internet. Understanding what is happening in these
environm ents is becom ing increasingly difficult. In other words, we need to find the
best ways to m ake sense of this wealth of data, to im prove the quality and
availability of inform ation, and to ensure effective responses.
An event is anything that happens [ Chandy 2009] . In an inform ation society, flows
of data can be seen as stream s of observable events. Event-driven system s provide
autom ation to allow the events to be interpreted and correlated, and support the aim
of delivering a tim ely response. Event processing is a set of techniques and tools that
help us understand and control event-driven system s [ Luckham 2002] . The key idea
is to explore tem poral, causal, and sem antic relationships am ong events to m ake
sense of them in a tim ely fashion. This reveals opportunities and threats as soon as
they em erge or can serve to diagnose and execute decisions in tim e constrained
fashion.
Most businesses today actively m onitor vast quantities of event data to m ake
autom ated decisions and take tim e-critical actions. Event processing provides real-
tim e visibility in a wealth of event data and enables responsiveness in decision-
m aking processes. A fundam ental characteristic of events is that they cannot be
entirely foreseen [ Chandy 2009] . Given that fact, we cannot predict when a critical
event will happen. What we can do is ensure that the response is provided with
m inim um latency. Tim eliness is one of the essential requirem ents in m any of the
event processing applications.
Event processing has em erged as a substantial new field of software engineering and
com puter science over the last ten years. It is now one of the fastest growing
segm ents in enterprise m iddleware software, with products provided by m ajor
software vendors and m any start-up com panies around the world. Given this
phenom enon, we exam ine why event processing is em erging now (and not som e
tim e before). We then classify categories of problem s that can be solved by event
processing and consider the typical com plexity drivers in each of these categories.
We exam ine the benefits of using this technology and the costs associated with it.
Finally, we discuss the buy-versus-build issue to provide som e guidelines for
successful adoption of this technology.
1 .2 W ho should read this?
This docum ent is directed toward several key stakeholders in event processing.
In particular, end users should read this to gain insight into the value that the event
processing approach provides to solving real business problem s. You will learn what
the unique characteristics are that m ake a problem ideally suited to being solved by
this approach as well as the costs and benefits of using this approach, as com pared
to alternate approaches.
We also address the broad applicability of event processing and explore the criteria
for and benefits of applying its approach to distinct problem types. Architects and

8. 8
analysts can use this docum ent to help enable them to recognize cases in which the
event processing approach should be applied.
1 .3 Origins of event processing
The concept of event processing is older than com puting itself. Doing work in
response to events, whether it is the receipt of postal m ail, the ringing of a fire
alarm , or the declaration of war, is a natural part of our society and econom y. But
recent years have seen a dram atic increase in the num ber of observable events, in
the need for tim ely responses, and in the autom ation of those responses. The
autom ation of m arkets, factories, and com m unication m eans that events that were
previously inaccessible to com puting are now available for processing.
Electronification has also led to people expecting m ore tim ely responses to their
needs. A six- to eight-week delivery process or a three-day wait for telephone
activation no longer m eet people's expectations. To achieve tim ely responses that
are acceptable, operations that were previously accom plished by people and
paperwork are being im plem ented with software and electronic com m unication.
Com panies today are operating at a faster pace, so the ability to handle m ore
observable events in a short tim e is increasingly im portant. The am ount of available
event data is rapidly expanding as well, because of the decreasing costs and
increasing speed of com puters and networks and the unifying power of the Internet
and its com m unication standards [ Chandy 2009] . The end result of these changes is
the em ergence of m any new system s that are required to process potentially vast
quantities of data in a tim ely fashion and to m ake com plex autom ated decisions
based on the inform ation available to them .
Event processing is experiencing an evolution from bespoke, custom ized, ad hoc
designs into an established com puting paradigm , just as other technologies have
done before. Techniques and com puting architectures such as report generation,
client-server protocols, and web services all started with custom ized
im plem entations based on older technology. Over tim e, best practices and design
principles for these system s em erged and were integrated into specialized com puting
platform s, just as they are now for event processing. Today, a variety of established
event processing technologies exist, as do a range of open research problem s to be
resolved. But as these technologies are im proved upon, and problem s are explored,
m any com m on characteristics have em erged.
Event processing offers a standardized and optim ized way to im plem ent event-based
applications. While building system s that process events using traditional com puting
architectures is certainly possible, m eeting the requirem ents of m odern system s this
way can be difficult. Using event processing, organizations are able to react m ore
quickly and are able to handle m ore data and m ore detailed inform ation, thus
m aking better decisions based on m ore sophisticated logic.
1 .4 Problem s solved by event processing
To understand why organizations are selecting an event processing approach for
im plem enting advanced IT solutions, we m ust consider application areas that have
already benefited from em ploying this technology. We m ust also understand the
characteristics of the applications for which event processing technology has been
selected.
In application areas like production m onitoring and control system s, location-based
services, algorithm ic stock trading, or logistics control, we can observe a trend away
from the traditional client-server interaction m odel. In this m odel, the client pulls

9. 9
inform ation from a server toward a m ore loosely-coupled, event-based interaction
pattern, in which partner applications em it inform ation in an asynchronous push
m ode.
This shift toward m ore asynchronous push interactions is supported by an event-
driven architecture pattern. This pattern supports event transport and processing as
a program m ing paradigm . This applies to external interactions of a system as well as
to system -internal com ponent interfaces. This shift is illustrated in Figure 1.1.
A closer exam ination into the kinds of applications m entioned above reveals the
reasons for shifting from request-response m ode to event driven m ode.
1 .4 .1 Manufacturing Execution system s
Manufacturing execution system s (MES) are the heart of production processes in
various industries. Both discrete and continuous production processes interact with
the production m achinery directing the production proper. Event processing can be
used in MES system s to detect anom alies and determ ine if significant changes
relative to assum ptions that require re-planning and predict future events and states
have taken place. Event processing in MES system s functions as a subsystem that
should interact with the rest of the system . This space of production m achinery is an
event source em itting condition inform ation about production equipm ent and
processes asynchronously. To enable this interaction, an event processing layer can
be found in the m ajority of MES. This event processing layer is a natural fit for the
interface between the m anufacturing execution and the plant floor control system s.
Using a client-server interaction m ode, in which the MES queries production status
inform ation from the plant floor, would invert this natural role-split since the MES
would becom e the client for the plant floor control system would logically have to
play the server role. An asynchronous push interaction from the plant floor control
system s to the m anufacturing execution system allows for m aintaining the view that
the MES is the server, with the plant floor control system s being the clients who feed
Figure 1 .1 : Event- driven interaction

10. 10
inform ation into the MES. The asynchronous push m ode from the plant floor system s
to the MES appears to be the m odel of choice. Com pared to the client-server pull
m ode, it also saves one half of the client-server interaction cycle, which allows for
better perform ance of the event-based interaction pattern. Speed of reaction is
im portant in the MES context, as the plant floor system s operate at a m uch higher
speed than the MES. Therefore, a client-server interoperation m odel between the
MES and the plant floor control system s would im pose im plem enting server
capabilities in the plant floor system s, slowing them down considerably. So this
production execution space nicely shows why an event processing approach is
superior to a request/ response-based client-server architecture.
1 .4 .2 Location-based services
Another area for applying event-driven application architectures is location-based
services, in which location sensors, such as active radio frequency identification
(RFID) tags, m obile phones, and Wi-Fi enabled devices feed inform ation about their
spatial location into server-side system s. These location feeds trigger services
depending on the spatial location of the client, like notifying transportation status for
shipped goods through RFID signals, searching for a restaurant from a m obile phone,
or tracking goods in the supply-chain and feeding respective m anagem ent
applications, for exam ple.
Also in these applications, as in the production control space, im agining how the
event-driven inform ation push delivery m ode from the devices to the servers could
be inverted to a client-server based inform ation pull m ode is difficult. In a client-
server based inform ation pull m ode, the servers contact the clients to request
location and status inform ation. While in the m anufacturing space the production
m achinery are known event sources, in m any location-based services, the sensors
delivering location inform ation are not necessarily known a priori to the servers.
Thus, server-side location-based service applications typically interact with an a
priori anonym ous set of clients, which do not necessarily know the servers to which
they want to deliver inform ation. This inform ation-delivery-only m ode distinguishes
event-oriented location-based services from standard Internet applications in which
anonym ous clients access a server they know in order to retrieve inform ation.
1 .4 .3 Algorithm ic trading
Financial data system s, whether in m arkets and trading or in surveillance and fraud
detection, have dram atically changed over the last 20 years, due to new electronic
business processes and increased service expectations of m arket participants. Many
of the new business processes are fully autom ated through IT system s, for which an
asynchronous approach seem s to be the m ost natural—and in m any cases the only
possible—approach.
Algorithm ic trading relies fast decision m aking based on observations of m arket
activities. Large num bers of m arket prices for financial products are typically em itted
to all m arket participants sim ultaneously and with low latency. Successful m arket
participants need to react in (quasi) real-tim e, using digital trading system s
im plem enting their investm ent strategies. Therefore, the event-based inform ation
dissem ination im poses an event processing approach on m odern financial IT
system s, in which (quasi) real-tim e m arket analytics can hardly be im plem ented in
conventional client-server architectures. Recent years have shown that event
processing is the right approach for coping with the large am ounts of inform ation
pushed into the data clouds of financial m arkets.

11. 11
1 .4 .4 Defense intelligence
Over the last 50 years, m ilitary intelligence has transitioned from an inform ation-
poor environm ent, in which gathering inform ation was the chief concern, to an
inform ation-rich environm ent. Today, the num ber of sensors, satellites, and soldiers
is pervasive, and the need to present a tim ely, correct, and integrated view of the
inform ation they provide is a critical factor for effectiveness.
Figure 1.2 illustrates som e exam ples of application dom ains and sum m arizes the
type of functions for which event processing system s are enablers. This is
sum m arized in [ Etzion 2010] .
The five different functions from Figure 1.2 are described below, followed by an
exam ple of a system in which event processing could perform each function:
Active diagnostics: finding the problem based on sym ptom s, exam ple: network and
system m anagem ent
Real-tim e operational decision: reaction to event within tim e constraints, exam ple:
algorithm ic trading in financial m arkets
Predictive processing: predicting future events and proactively m itigating undesired
events and exploiting opportunities, exam ple: rerouting due to traffic problem s.
Observation system s: observe exceptional behavior (such as KPI threshold
breaching) and notify in various ways, exam ple: BAM system s
Figure 1 .2 : Types of functions in event processing

12. 12
Inform ation dissem ination: subscriptions to particular inform ation that can be
filtered, transform ed, or derived from stream ing events, exam ple: personalized
notification from financial data.
A particular application m ay intend to satisfy m ore than one of these functions.
1 .5 Criteria for adopting an event processing approach
An event processing approach is ideally suited for applications delivering situational
awareness and response, with a com bination of the following requirem ents:
The system (s) under control/ observation is event-driven, with a high volum e of
events representing accum ulating, increm ental state change
Tim ely action is required—in som e cases, that action m ust have the potential for
im m ediate effect
The application has a high degree of com plexity as m easured by any of the following
factors:
1 . Degree to which the application is expected to change over tim e, e.g., with new
event sources, new interactions and new responses expected
2 . Num bers and types of event sources
3 . Num bers of consum ers of inform ation com m unicated in the events
4 . State and context m anagem ent
5 . Opportunity to create new value, e.g., by introducing reflection and introspection
The event data can be m ade available in electronic form
There is a continuous evolution of requirem ents, e.g., new event sources/ types or
new response opportunities
Som e applications that do not m eet these criteria m ay also benefit from this
approach, but the benefits m ight be less significant.
Event volum es and tim eliness are two factors that m ost typically influence the
selection of an event processing approach. However, other com plexity factors can
m otivate this decision as well. In Table 1.1, we sum m arize this in a sim plified
decision table that shows areas in which event processing can be particularly
applicable.

13. 13
In this table, event rates refer to the rate of input events to the system s. Application
com plexity refers to the five criteria listed above, and tim eliness refers to the level of
im portance of the reactions m eeting tim e constraints.
Event
driven
nature
Event
Rates
Application
com plexity
Tim eliness
Recom m ended
for event
processing
Exam ples
High
High High High Yes
Advanced
algorithm ic
trading
Medium /
High
High High Low
Depends on the
type of
com plexity and
the secondary
benefits of
im proved
tim eliness
Supply chain
m anagem ent
(SCM), fleet
m anagem ent
High High Low High Yes
Order
routing
Low/ Medium High Low Low
Lim ited event
processing
capabilities such
as filtering m ay
be sufficient
Possibly
suited to
business
activity
m onitoring
High Low High High Yes
Earthquake
detection
using
distributed
low cost
sensors
(low average
rate, high
bushiness)
Low Low High Low
No; Com pute-
heavy
transaction
Low Low Low High
No; Use
m essaging
system
Low Low Low Low No
Table 1 .1 : Event Rates, tim eliness, and other com plexity drivers

14. 14
Event-driven applications som etim es need to process large volum es of data and
enable appropriate reactions using sophisticated logic. The following m easures of
com plexity also m otivate an event processing approach:
Degree of im portance of tim eliness: The effectiveness of a response depends on its
tim eliness. For exam ple, a tsunam i warning issued after a tsunam i strike has little
value. Event processing is a valuable technology in applications in which the value of
a response to an event decays with increased response tim e. The m ore rapid the
decay, the higher value of the technology. Certain applications (e.g., algorithm ic-
trading applications) would not exist today if a high-value response was not related
to tim eliness. On the other hand, event processing is also valuable in applications in
which tim eliness is not the m ost critical com plexity driver. Therefore, exam ining
other com plexity drivers is also worthwhile, as discussed below.
Intensity of event data volum es: Event processing provides intelligent analysis of
stream ing data (events) in a tim ely fashion. As the am ount of available data in
digital form is rapidly increasing, the challenge becom es yielding the intelligent
analysis on larger volum es of data. Event processing offers concepts and tools that
provide on-the-fly analysis on large volum es of data.
Frequency of application requirem ent changes: In a typical lifecycle of a business
application, its requirem ents frequently change. The changes happen due to the
need to m eet new custom ers’ requirem ents or new governm ent regulations. For
whatever reason, business requirem ents change, and the com plexity of the
application m ay also increase with these changes. Event processing system s offer a
high-level abstraction layer, in which changing an existing event pattern or adding a
new one m ay significantly influence businesses strategy. Hence, frequently changing
event patterns provides a m eans to cope with frequent requirem ent changes.
Num ber of event types: Handling interactions am ong different parts of an
application, or interactions am ong distributed applications m ay be a tedious task.
Com m unications of one-to-one or one-to-m any m ay be also hard to m anage. In
event-driven system s, different interactions and com m unications are m odeled with
different event types. By publishing definitions of event types to all interacting
com ponents, we can effectively increase transparency of the com m unication and
ease the application integration. Moreover, we im prove m anageability of com plex
applications. We can also respond to rapid changes of interactions by changing event
types.
Num ber of consum ers of events: Publish/ subscribe is a m essaging paradigm in event
processing, in which senders (publishers) of m essages are not program m ed to send
their m essages to specific receivers (subscribers). Publishers are loosely coupled to
subscribers and need not even know of their existence. This m echanism m ay
therefore enable greater scalability in applications in which a large num ber of event
consum ers (subscribers) exist, as well as in applications in which a m ore dynam ic
network topology am ong consum ers is required.
State and context m anagem ent or rule interdependencies (including com plexity of
queries): Real-tim e applications with m any interactions are difficult to understand
and program . Event processing is a set of techniques m eant to help us understand
and control such system s. It provides abstractions to cope with these com plexities.
These abstractions are used for capturing business situations that we want to
m onitor. That is, they enable a business user to express high-level business goals,
hiding away the com plexity of the event-driven interactions.
Opportunity for the introduction of new value, e.g., reflection, introspection, or
retraction: Having an event as a first class citizen in a real-tim e program m ing m odel
opens new horizons and challenges. For exam ple, we can m ake new-generation
applications dealing with the following: inexact event processing (i.e., uncertain data

15. 15
stream processing), out-of-order event processing (i.e., processing of delayed
events), introspection (i.e., inspecting why a certain business event happened and
why another, expected one, has not), event retraction (i.e., retracting erroneously
recorded events and com puting revisions on detected com plex events), and so forth.
In general, the presence of these factors renders the event processing approach
beneficial. Event processing has evolved to address these kinds of challenges. Tools,
analysis, and m odeling m ethodologies have been developed, and together with event
processing run-tim e environm ents (m iddleware), a com pelling case for event
processing is em erging.
1 .6 Benefits
Enterprises use event processing to im prove the perform ance of their business in
m any ways, including increasing revenue, lowering costs, reducing risk, and
com plying with regulation. The essential role of event processing is to provide
inform ation that leads to faster and better decisions, as com pared to decisions based
on the use of traditional applications, m anagem ent reports, or business intelligence
(BI) initiatives. Event processing enables sense-and-respond behavior, in which
incom ing inform ation is used to sense the current situation and a person, device, or
autom ated com ponent responds in a tim ely fashion.
1 .6 .1 Event processing system s provide inform ation faster
Such system s act on a new event as soon as it arrives. The receipt of event data
triggers com putation im m ediately, in contrast to traditional applications and business
intelligence system s, which are either tim e-driven or request-driven. Tim e-driven
and request-driven system s store data when they arrive, and processing is triggered
later by a clock (in a tim e-driven system ) or by a request from a person or com puter
program (in a request-driven system ). This postpones the com putation and therefore
defers the response.
Event processing software m inim izes processing tim e by keeping m ost data in
m em ory and using data structures that are designed to facilitate fast access. Many
calculations are increm ental. Event processing system s also im plem ent other
strategies to m inim ize excess code path, context switches, and other overhead.
1 .6 .2 Event processing system s im prove the quality of available
inform ation
Event processing system s com bine inform ation from m any data points to generate
new insights. They also use algorithm s and rules to process event data that they
have received from one or m ore sources during som e tim e period. The system s also
generate sum m ary-level facts (com plex events) and put them in context to identify
threats and opportunity situations. Event processing system s apply a variety of
techniques that are particularly appropriate for handling event data. These
techniques include provisions for dealing with tim e windows, such as events that
occurred within a few m inutes of one another, or events that occurred within the
m ost recent few seconds, m inutes, or hours. Event processing system s have ways to
identify events that cause other events to happen (causality).
As a rough guideline, alerting system s, reports, and dashboards that m ust
recalculate in ten m inutes or less are likely to require event processing approaches.
Applications that run less frequently can generally be im plem ented by traditional,
periodic IT architectures. However, even som e of those applications would run faster,
consum e less com puter resources, and be easier to develop if they were

16. 16
im plem ented using continuous event processing, particularly if they deal with
tem poral or causal relationships am ong the event data.
1 .7 Benefits of event processing system s
The speed and inform ation-generating capabilities of event processing system s lead
to four kinds of benefits:
Operational changes can m ade sooner, m aking them m ore effective
Radically new, tim e-sensitive business practices are m ade possible
Better inform ation leads to better decisions
Staff cost is reduced by offloading parts of work to com puters
These are explained in the subsequent four sections.
1 .7 .1 Operational changes can be m ade sooner, m aking them
m ore effective
In m any aspects of com pany operations, m anagers can achieve an econom ic benefit
by fine-tuning the way work is done to m ore closely m atch current conditions.
Com panies reduce their costs or increase their revenue by im plem enting
m anagem ent decisions earlier, rather than m aking the sam e or sim ilar changes later
(see Figure 1.3).
For exam ple, custom er contact centers were traditionally m anaged by m eans of daily
reports. These reports provided statistics on call volum e, custom er on-hold wait
tim e, call duration, transferred calls, dropped calls, and whether the custom ers’
issues were resolved in the first phone call. Contact center m anagers and supervisors
used the reports to decide whether to bring in m ore agents, adjust staff
Figure 1 .3 Value of having operational m anagem ent inform ation sooner

17. 17
assignm ents, m odify the call scripts, or change other aspects of their day-to-day
operations. Over tim e, contact centers began m oving to hourly reports to enable
m ore frequent, intra-day adjustm ents, rather than m aking the changes on a daily
basis. If call volum e and wait tim es grew, agents on outgoing calls could be switched
to incom ing calls, rem ote agents could be added to the available work force, or other
aspects of the operation could be m odified. Idle agent tim e was reduced and
custom er service levels were enhanced, but further im provem ent was still necessary.
Today, m any contact centers have m oved to continuous m onitoring, based on event
processing. Sophisticated custom er relationship m anagem ent (CRM) analytic suites
incorporate real-tim e dashboards with graphical displays that update every m inute,
providing near-real-tim e visibility into the state of the contact center. Supervisors
can spot problem s as they em erge, analyze root causes, and quickly m ake changes.
This m inim izes wasted tim e and staff cost and m axim izes custom er service and long-
term com pany revenue. If data indicate that an individual agent has had several
problem atic calls in a row, a supervisor can be alerted to give the agent an early
break to recover.
Other exam ples in which event-based m onitoring system s im prove the tim eliness of
operational m anagem ent decisions include airline operations and supply chain
m anagem ent system s.
1 .7 .2 Radically new , tim e-sensitive business practices m ade
possible
Event processing enables som e radically new business processes that were
previously im practical because the necessary inform ation could not be obtained in a
tim ely m anner. Event processing system s can collect and analyze a large body of
data relevant to a current situation fast enough to affect individual transactions still
in process.
Figure 1 .4 Value of inform ation to affect individual transactions

18. 18
For exam ple, agents and supervisors in a custom er contact center can use
inform ation derived from event data to adjust their treatm ent of a custom er while he
is still on the phone. An agent dealing with a custom er who has been transferred
several tim es or has been on hold for m ore than five m inutes can be prom pted (and
allowed) to give additional com pensation for the negative experience. The contact
center software m ay also generate screen pops, m essages that prom pt agents to ask
custom er-specific questions to drive up-selling or cross-selling.
Alternatively, a supervisor could becom e involved in a call if an agent is struggling.
System s m ust generate alerts within a few seconds if the goal is to enable
intervention in a call that is still in progress (see the line plot on the right in Figure
1.4). After 30 seconds, the window of opportunity to affect that transaction has
closed.
Sim ilar kinds of im provem ents are found in fraud-detection system s, custom er
experience m anagem ent system s in gam bling casinos, and m obile telephone
provisioning system s.
High frequency and other algorithm ic trading system s in capital m arkets are even
m ore tim e-sensitive. Som e trading strategies require that buy and sell decisions be
m ade within a few m illiseconds, because the opportunity would be gone when prices
change slightly, or when a com petitor has grabbed the deal. In such cases, m erely
perform ing event processing well is not enough—the com pany m ust do it faster than
its com petitors. A particular calculation can be worth hundreds of thousands of
dollars if action is taken within a few m illiseconds but be worthless 10 or 20
m illiseconds later (see the line plot on the left in Figure 1.4). These system s m ust be
fully autom ated; involving a person in an individual trade could not be done in tim e.
In fact, a person takes 100 m illiseconds just to type one character into a keyboard.
Sim ilarly, to be effective, som e m ilitary weapons and defense system s m ust also
operate using such extrem e response tim es.
1 .7 .3 Tim ely inform ation dissem ination leads to better
decisions
The inform ation provided by event processing system s is often im possible or
im practical to obtain through any other m eans. Event processing system s can quickly
extract and distill the inform ation value from dozens, thousands, or even m illions of
data points. From these data, EP system s can produce alerts, key perform ance
indicators, or other m etrics to aid in decision m aking. By contrast, a person, without
the assistance of EP system s, can directly assim ilate only a few data points at a tim e,
and thus cannot consider nearly as m any factors when m aking a decision.
For exam ple, com panies that m anage large fleets of trucks get raw inform ation
about the location of all of their trucks through GPS system s. Basic fleet
m anagem ent system s can display truck locations on m aps so dispatchers can know
the location of each truck. However, such m assive volum es of data can m ake
spotting situations that require attention difficult. Event processing system s can
easily com pare each truck’s location against its planned itinerary and raise alerts for
the few trucks that are far from their expected locations, or for trucks that are within
m inutes of being ready to load, unload, or require som e other kind of attention.
Event processing system s are typically used to im plem ent m anagem ent-by-exception
strategies. They reduce the volum e of unwanted data, known as inform ation glut,
presented to people. In som e cases, a system m ay run for hours or days, turning
m illions of base data points into thousands of com plex events before detecting a
single com plex event that m ust be brought to the attention of a person. People are

19. 19
disturbed less often, so they can reserve their attention for the few situations in
which their involvem ent is im portant.
The need to deal with high volum es of current event data is em erging in m any
applications and industries, including the following exam ples.
Track-and-trace system s can report the history of pharm aceuticals from factory to
patient to com bat theft and drug counterfeiting and to conform to regulations. Such
system s m ay save detailed event history data for m illions of packages, for m onths or
years. They can retrieve and correlate the data when a question about a particular
item or a batch of goods is raised.
Context-aware, location-based services in m obile telephone networks use event
processing to track the locations of hundreds of thousands of subscribers
sim ultaneously. The system can notify a person if som eone in her group is nearby,
by finding m atches am ong m illions of data points.
Network security system s use event processing to detect patterns that indicate
denial-of-service attacks or various form s of fraud, such as poaching and spoofing
user I Ds or theft of service.
Package delivery services scan m illions of packages several tim es per day using bar
code or RFID readers to m onitor and report the history, current location, and
projected delivery tim e of every box or envelope.
In all of these exam ples, the inform ation value in each individual data point m ay be
sm all, but the value of accurately distilling a large am ount of data into a few
significant, com plex events is enorm ous. Com puters do not succum b to boredom , so
they m ake fewer m istakes than a person (even given that he had the tim e and
ability to do the com putations). Com puter calculations are consistent and repeatable,
because they always im plem ent the sam e algorithm s in the sam e way, as defined by
knowledge workers, analysts, and software developers.
1 .7 .4 Staff cost is reduced by offloading w ork to com puters
Som e applications can be done fast enough and well enough by hum ans, but can be
done at a lower cost by event processing system s. EP system s offload the drudgery
of repetitive calculations and pattern detection com parisons from people to
com puters in m any scenarios, including in the following two exam ples.
The spread of algorithm ic trading system s has reduced the num ber of hum an traders
operating on certain categories of investm ents in capital m arkets. Traders still
form ulate the strategies and rules that are executed by the trading system s at run
tim e, and traders also still carry out m ore com plex, dynam ically-determ ined trades,
and m onitor the perform ance of the autom ated system s.
The num ber of people m onitoring the operations of airlines, shipping fleets, and
trucking com panies has also been cut in som e areas, because EP system s enable
each person to track m ore vehicles.
The am ount of decision m aking that should be offloaded to EP system s varies
depending on the business problem . Managers and knowledge workers can delegate
as few or as m any tasks as they see appropriate to the software.
1 .8 Build-versus-buy
Build-versus-buy rem ains a m ajor issue in the event processing m arket. Any event
processing application can be im plem ented with standard program m ing languages
and tools, and before the em ergence of com m ercial event processing platform

20. 20
products, all applications that needed EP capability used application-specific (build)
code to do so. Build EP logic was still used in about 95% of EP applications in 2010,
because m ost event processing applications have relatively infrequent changes and
m odest requirem ents regarding throughput, latency, and rule com plexity. However,
this is changing, as organizations begin to recognize the non-linear rise in costs as
change m ust be introduced, and begin to appreciate the com petitive advantage that
tim eliness (and other differentiators) can deliver.
This tradeoff can be considered in light of three stages of m aturity—m anual, build (or
adapt), and buy (e.g., using existing EP technology), as illustrated in Figure 1.5.
The curves represented in this figure are actually quite generic and can apply to any
new technology and accom panying services capability. The purple curve represents a
m anual approach, m eaning without the use of EP system s, and the green curve
represents the build option, while the blue curve represents the buy option. The
curves reflect the non-linear increase in costs associated with rising com plexity. They
acknowledge that adopting a m anual approach for sim ple applications (hiring m ore
people to m onitor telephones or com puters, for exam ple) m ight be easier than
adapting an EP system , but that at som e point, the introduction of new challenges
will render a m anual approach unaffordable.
Such challenges could be in the form of higher event rates or m ore event types. A
build approach m ight be considered as the first upgrade from a m anual system , in
which custom software is developed, but again, the com plexity of m anaging that
stack and the probable lack of flexibility to address new opportunities will eventually
have the sam e result. An enterprise should consider m oving to the next phase when
the com plexity rises and m akes the upgrade cost-effective.
Figure 1 .5 . Cost vs. com plexity in use technologies

21. 21
We believe that m ore dem anding, high-volum e/ low-latency applications are
increasingly likely to use the buy approach, nam ely by em ploying com m ercial-off-
the-shelf (COTS) event processing technology
Table 1.2 illustrates som e considerations in the build-versus-buy decision.
Build Buy
Pro
• Purpose-built
• Sim plicity
• Lower learning
curve
• Lower tim e to
benefit
• Greater flexibility
• Tim e to change is
lower
• Existence of
Support/ Tools/ Servi
ces
Con
• Cost of
developm ent if the
application is
com plex
• Lack of flexibility
• Higher risk
• No services
available
• Cost of acquisition if
the application is
sim ple
• Cost of integration
to existing system s
• Vendor risk
Table 1 .2 : Build- versus- buy table
1 .9 Sum m ary – the value of event processing
This section surveys som e considerations in answering the following questions:
When does it m ake sense to use an event processing approach?
In which cases does using generic event processing software suffice?
The m ain factors that support using the event processing approach are:
Use of an application that includes som e event-driven requirem ents (a response is
required following certain events)
The applications event-driven requirem ents are relatively com plex
Tim ing constraints apply to the required responses
Com plexity is often the crucial factor, since satisfying requirem ents using
conventional program m ing abstractions m ay be difficult. But in som e cases, the
tim eliness of reactions is an im portant consideration as well.

22. 22
Chapter 2 : W hat are the characteristics of
event processing?
This chapter exam ines m ore deeply what we actually m ean by the term event
processing. In the following sections, we expand on our general definition of the
term and address the kinds of problem s that event processing can be used to solve.
We then describe a set of characteristics com m on to these event processing
applications.
2 .1 W hat do w e m ean by event processing?
We start with som e term inology.
Event processing: This can be broadly defined to be any com puting that perform s
operations on events. Com m on event processing operations include reading,
creating, transform ing, and deleting events. They can also include the distribution
and dissem ination of events am ong participants in a distributed com puting
system . The term can refer to specific software or hardware that does the
processing, or to the subject area as a whole. Processing of events is usually done
with the purpose of m eeting som e kind of application goal, leading to our next
definition.
Event processing application: This is an application of event processing to solve a
particular problem , or m ore generally, any com puter application that em bodies
principles of event processing.
Event processing platform s: These refer to system s and tools that help develop, run,
m anage and m aintain event processing applications. These are frequently used to
take som e of the burden of program m ing event processing off of the author of the
application.
Event processing involves two m ain ideas:
Processing events to gather m eaningful or valuable inform ation and then
deriving actions from them :
The events in question usually represent som e aspect of som ething that occurred in
the real world. The significance often arises from the com bination of m any events,
rather than any single event, and tim e is often im portant. The inform ation conveyed
in a set of events often depends on the order in which they occurred, and/ or the tim e
at which they occurred
The value of inform ation m ay well depend on the tim eliness with which it can be
supplied to a consum er, varying from consum er to consum er and depending on
external circum stances, such as the state of the overall system .
Dissem inating or distributing the events:
This idea refers to capturing events and then transporting them to the consum ers
who need them , ensuring that the appropriate interm ediate processing is perform ed
on them . Dissem inating and distributing is also about getting the right inform ation to
the right consum ers at the right tim e.
The distribution process serves to decouple the event producers and the event
consum ers, so that event producers do not need to be aware of the physical
addresses/ identities, or even the existence, of the consum ers. In m any cases, the
consum ers are not concerned with the identities or addresses of the producers. This
decoupling (often achieved using a publish/ subscribe paradigm ) allows event
processing applications to evolve over tim e, in a flexible way. Further event

23. 23
producers, consum ers, or event processing functions can be added without
disrupting existing users or functionality.
As well as transporting event m essages, an event distribution system m ight need to
provide adapters. Adapters allow other applications to act as event producers or
consum ers. Distribution system s also frequently provide m echanism s to advertise
the availability of events or to allow consum ers to register their interest in particular
event types.
A large scale event processing application m ight involve thousands of event
producers or consum ers, dispersed over a wide geographic area, as in a track and
trace application m onitoring a supply chain, for exam ple. Such an application m ight
have to handle tens of thousands of events per second. So som e applications need
an event distribution system that can scale to handle high event volum es and large
geographic areas, and possible to also span m ultiple types of com m unication
technology—private networks, public internet, wireless m obile networks, or wireless
sensor networks, for exam ple.
2 .2 Characteristics of event processing applications
The rationale for event processing is that it facilitates the design and im plem entation
of a certain class of com puting applications, which we call event processing
applications. So what exactly is an event processing application?
First and forem ost, these applications are centered on the ideas of events and
actions. Such event can be ingested from the outside world, generated by the
application itself, or derived from the processing of these events. Applications are
frequently event-driven, in that events are pushed through the system and are
processed as soon as, or soon after, they occur. In som e cases, events m ay be
stored in a database or log for subsequent retrieval and further processing, but the
initial processing is still im m ediate.
The way in which events are processed in an event processing application is
influenced by the context in which the events occur. In m any applications, the
tem poral context—either the absolute occurrence tim e, or the ordering of events
with respect to one another, or both—is very significant.
As we noted in the previous section, event processing applications m ay involve large
num bers of event producers and consum ers, which are often distributed over m any
system s. These system s m ight have differing com m unications capabilities and m ight
be dispersed into widely different geographic locations. An application m ay also need
to distribute its processing of events over m ultiple m achines to achieve throughput
and latency objectives. As such, event processing applications can be heavy users of
concurrent processing techniques.
Som e event processing applications are highly dynam ic. Event volum es m ay vary
over tim e, and the nature of the processing m ay vary depending on the interests of
the consum ers or the nature of the incom ing events. In addition, event producers
and consum ers m ay com e and go during the execution of the application.
2 .3 Event processing application requirem ents
We now enum erate the m ain functional capabilities required by event processing
applications. Event processing applications com e in m any shapes and sizes, and as
such, not all applications will necessarily use every capability listed in this section.
Also, a capability can be im plem ented either natively by the application, or provided
for that application by an event processing platform .

24. 24
2 .3 .1 Event I nput/ Output
Most applications need som e way to ingest events from external event producers.
These could be hardware sensors, software probes, external data stream s (such
as stock prices) or news feeds, event logs, or files of pre-recorded events.
Sim ilarly, applications need a way to output events to external consum ers. This
can be perform ed by actuators, business applications and processes, and event
logs, for exam ple.
2 .3 .2 Data Reduction
The process of extracting inform ation from events often involves techniques that
reduce the am ount of data contained in the ingested events. These techniques
include:
Filtering: This refers to elim inating entire events as un-interesting, either by nature
of the event or according to som e other m etadata associated with the event (such as
its producer) or to the values of attributes contained in the event. Som e applications
use sam pling filters, which select a subset of incom ing events, to achieve a particular
data rate without regard to other criteria.
Projection: This m eans reducing the size of an event by discarding som e of its
attributes (for privacy reasons, for exam ple).
Aggregation: This refers to com bining a set of events by extracting one or m ore
attributes from each and perform ing som e kind of com putation on these attributes
(averaging or calculating the m inim um or m axim um , for exam ple).
2 .3 .3 Reasoning:
The techniques used to extracting derived events out of the raw events include:
Transform ation: This is the process of taking an event and producing a new one
that presents m ore m eaningful inform ation. This inform ation can be derived by
analyzing the data contained in the original event (such as car license plate
recognition), or it can be derived with the help of som e external data source (a
database of custom ers, for exam ple).
Aggregation: In addition to their role in data reduction, aggregation operations can
be used to derive additional inform ation (counting event arrivals, for exam ple) or to
elim inate errors and noise from incom ing events.
Pattern detection: This is a type of processing that exam ines a collection of events
and looks for m atches or other patterns am ong them . The presence of such a pattern
often signifies som ething of m ore im portance than the raw events them selves. A
special case of pattern detection is absence detection, in which the fact that a given
event did not occur in a particular tim e period is of itself significant.
2 .3 .4 Context aw areness
The way in which events are processed, including the data reduction and
reasoning operations we have just m entioned, often requires taking into account
the context in which the events occurred. This context can involve one or m ore of
the following:
Segm entation using A key attribute (or com bination of attributes) of the event;
This is frequently used to identify an external entity to which the event relates, for
exam ple, the ID of a patient wearing a health m onitor, or the stock ticker sym bol in
a trading application. We refer to this as a segm entation context.

25. 25
The state of an entity external to the event processing application: This could
be an entity referred to directly by the event. We would expect, for exam ple, a
person’s heart rate to be different depending on whether she is resting or
exercising. This could also refer to som ething m ore general, such as the weather
conditions where the event occurred or the state of the business in a business-
oriented application.
The location at w hich the event occurred, or its proxim ity to other related
events
The tim e at w hich the event occurred, or its tem poral relation to other related
events
2 .3 .5 Logging and analysis
Applications frequently keep a record of som e or all of the events they receive
and of the derived events they generate. This can be used for audit purposes, for
retrospective event processing (revisiting precursor events when exam ining a
problem , for exam ple), or to allow offline analysis.
Logging: A history store keeps a log of events that can be later queried.
Provenance tracking: This extends the history store to include inform ation about
how each logged event cam e to be produced. I n particular, if the event was one that
was derived by the event processing system itself, this inform ation could include the
events and derivation process that gave rise to them .
Visualization of event data: Most event processing applications include som e kind
of visual display of raw and/ or derived events. This can be the prim ary purpose
of the application, as seen in m any m onitoring or custom er-facing track and trace
applications, or it can be a tool used in developm ent, testing, problem
determ ination, or operational m anagem ent. Business activity m onitoring and
network and system m onitoring tools provide sophisticated dashboards for
display of visual inform ation. Map-based m ashups are a popular way to display
location-specific events.
2 .3 .6 Prediction
Event processing applications are som etim es used to m ake predictions about the
future.
Predictive pattern detection: This can be used to look for specific patterns that
indicate when an event is likely to occur. In particular, applications can be used to
detect anom alous patterns that suggest a problem is likely to occur. For exam ple, a
traffic m anagem ent system that could spot that gridlock is about to occur is m ore
useful than one that can only detect that it has already occurred. The patterns used
(and any param eter thresholds contained in them ) have to be carefully chosen to
avoid too m any false positives or false negatives.
On- line scoring: Events, or collections of events, are pre-processed using the
techniques we have already addressed and are then passed to a scoring engine,
which runs them against an appropriate data-m ining m odel. This process can be
used to assign a probability rating to the occurrence of a future event.
Sim ulation: Another way to predict the future is to use a com puter m odel that
sim ulates the dom ain being studied and to feed real-life events (after som e pre-
processing) into that m odel.

26. 26
2 .3 .7 Learning and Adaptation
In m any applications the operations perform ed on events are specified by hum an
beings, either by the program m ers responsible for designing the application, or
by users of the application who supply rules specifying som e or all of the
processing to be perform ed. However, m achine learning and sim ilar techniques
are being increasingly used for event processing, particularly in applications used
for m aking predictions.
Pattern discovery refers to the autom ated generation of the event patterns to be
looked for by subsequent pattern detection. One approach is to run offline analytics
against a log of stored events
Building a scoring m odel: Applications that use online scoring need a m odel for
that scoring. These m odels can be built by feeding them events from the event
processing system and can be adapted over tim e by feeding back predicted and
actual results. This will reduce the incidence of false positives and false negatives.
Application evolution: Event processing applications often m ust evolve over tim e.
The volum e and the nature of raw input events available to the application can
change. In addition, the required processing, such as the patterns being detected,
can vary—either in response to new user requirem ents, or in response to situations
being detected by the EP application itself.
2 .3 .8 Distribution
Last, but not least, event processing system s m ust ensure that events are
transported from the places where they are detected to the place or places where
the event processing occurs. EP system s m ust also ensure that any outcom es from
that processing are delivered to the right places.
Routing: This m eans ensuring that incom ing events are routed to the right
processing logic, and that output events are routed to the consum ers of those
events. In sim ple cases, this routing is static, m eaning that the routes are defined
when the application is developed and only change if the application is explicitly
m odified. In contrast, dynam ic routes can be created, deleted, or m odified while the
application is running, either as a result of user interaction or as a result of situations
detected by the application itself.
Partitioning: Event processing can itself be distributed across m ultiple servers. One
com m on approach is to partition the work across a horizontal cluster of servers and
to route events to the appropriate server. In addition, processing such as data
reduction can be perform ed as part of the event distribution process. This allows this
function to execute close to where the events are detected, reducing overall network
traffic.
2 .4 Non-functional requirem ents
Non-functional requirem ents are concerned not with what the event processing
application or platform has to do, but rather with describing how well it needs to do
it. These requirem ents apply both to the event processing and to the event
distribution aspects of an event processing application. They can be grouped into the
following areas:
2 .4 .1 Perform ance
Perform ance requirem ents can relate to an entire flow through an event
processing application, or just to a particular part of it.

27. 27
Response tim e m easures the tim eliness, or latency, of the event processing and
distribution. You m ight be concerned about the tim e taken between detecting an
event and displaying it on a dashboard, or the tim e taken to detect a particular
pattern. Response tim e requirem ents can be expressed in term s of:
• An upper bound of acceptable response tim e
• The average desirable response tim e
Predictability: A variance of other m easure of predictability—in som e applications,
predictability and consistency of the latency for all events is m ore im portant than the
overall speed, if a high average response tim e would m ean that som e response are
very fast and som e are slow
Scheduling of w ork and delivery of results: Multiple consum ers using an event
processing application can be in contention with one another. The system m any need
to assign priority to its workloads, to ensure that critical processing is not delayed by
less im portant work. In som e cases, such as trading applications, fair treatm ent of a
large group of consum ers, m eaning that no one consum er gets priority over others,
is critical.
Throughput: (often expressed as events per second) specifies the rate of incom ing
events that the application m ust be able to handle, without com prom ising response
tim e or other requirem ents.
Scalability and Elasticity: Scalability is the ability of a system to handle growing
workloads in a graceful m anner, possibly with the provision of additional hardware or
other resources. Elasticity is the ability of a system to scale up or down without
requiring application redesign or significant operator intervention. Event processing
workloads include:
• The num ber of event producers and consum ers;
• The com plexity of the processing to be perform ed—such as the num ber of
operations, and the frequency at which they execute;
• The num ber of output events (actions) that are produced (Consider a system
that is perform ing pattern detection (if the patterns being looked for occur
frequently in the input events, then the system workload will be higher than
in a system in which m any of the input events can be quickly filtered out;
• The input event throughput and the com plexity (including size) of the input
events;
• The am ount of event and other data that needs to be stored
2 .4 .2 Availability and Recoverability
Event processing applications typically run for extended periods of tim e, so the
ability to sustain long term work loads is often im portant. This includes:
Tolerance of hardware and software faults and network failure
Ability to recover the system after a failure or m ajor disaster
Continuous operation: the ability to effect a planned change to the application
while it is still running

28. 28
2 .4 .3 Consistency and I ntegrity in a distributed system
Perform ance and scalability requirem ents, and the sim ple fact that event
producers and consum ers are often located in different places, m ean that m ost
event processing applications involve m ultiple servers and m ultiple threads of
execution within each server.
Many of the operations used in event processing are influenced by the tim e at
which events occur, or by the relative order in which they occur. Im plem enting
these operations in a distributed system involves som e challenges. First, we m ust
define what we m ean by sim ultaneity. Second, we m ust address the issue of
com m unication delays (including network breakages) when events are detected
by different servers. Third, we m ust address the non-determ inism that is
introduced by the use of m ultiple parallel processing threads.
Different applications have different requirem ent in these areas, including the
following:
Tem poral granularity: This is the precision to be used for tim estam ps and tim e
com parisons.
Repeatability: This requirem ent addresses the issue—if two copies of the event
processing application were to be running side by side against the sam e set of input
events, the sam e set of outputs should be generated in both cases.
Consistency: When m ultiple event consum ers exist, is there a requirem ent that the
events sent to one consum er should be consistent with those sent to the other.
Relevance and Quality of the output events: The usefulness of an event
processing application depends on the quality of the events that it produces. This can
be influenced by m any factors, including the following:
• The precision and accuracy of the input events
• The event processing logic used by the application, particularly when the
application is perform ing prediction
• The availability, consistency and integrity issues m entioned above
• The location at which the application is being used to detect or predict
anom alous situations is an im portant m etric is the rate of false positives or
false negatives. In addition, som e applications provide a probability to the
events that they derive.
2 .4 .4 Security and privacy
Event processing system s are frequently used to m ake decisions that have a
tangible effect on a business or on a wider com m unity, thus, protecting them
against cyber-terrorists or other subversive attacks is crucial. This process
involves:
Controlling event producers access so that only authorized parties can be
producers of events Checking the events subm itted by these authorized parties to
m ake sure that they are only introducing events to which they are entitled to do
Controlling application logic access to ensure than only authorized users can
specify event processing logic and configure the system
These system s som etim es have to handle confidential data, including sensitive
personal inform ation, and in som e cases, the application is used to derive further

29. 29
inform ation about individuals. This involves a further set of requirem ents,
including the following:
Controlling consum ers access so that only authorized parties can be consum ers
of events
Anonym ization: Providing the ability to anonym ize or rem ove sensitive data fields
Auditing: Providing auditable logs of the processing perform ed and data transm itted
to third parties
In addition, the standard requirem ents expected of secure com puting system s also
apply, including the following:
Resistance to tam pering: Ensuring that com m unication links are resistant to
tam pering and providing confidentiality where needed
Data protection: Ensuring data persisted to storage m edia is protected
Code protection: Ensuring that the system code and any application logic are
protected from unauthorized access
2 .4 .5 Usability/ Maintainability/ Manageability
Many im portant considerations fall under this heading. First, requirem ents on the
languages and tools m ust be used to define the event processing logic, as
follows:
Usability: The authoring tools m ust appropriate to the kinds of user who will be
defining or m aintaining the logic, be they IT professionals, engineers, m edical staff,
business m anagers, the general public, or anyone else.
Versatility: The authoring tools m ust allow contributions from m ultiple stakeholders
with different backgrounds.
Expressiveness: The language needs to scale to allow com plex applications to be
developed and for these applications to be m aintainable, including the tools used to
help check the correctness of applications.
Maintainability: In addition, event processing platform s m ust be m anageable from
an operational perspective, particularly as m any event processing applications run
for an extended period of tim e. Considerations include: Ease of adding new
producers, consum ers, processing logic or servers; Ease of m anaging the security
aspects of the system ; Monitoring the running of the system itself, and the event
producers that are feeding events to it; Tools to help diagnose and fix problem s; and
disaster recovery processes and tools.
Tradeoffs m ust often be m ade am ong the objectives listed in this section. In som e
cases, a tradeoff of these objectives against the function requested of the application
m ust be m ade. Adaptive load-shedding is an approach that balances these
objectives. In this approach, an event processing platform stops perform ing less
im portant event processing operations to ensure that it can m eet the perform ance or
other requirem ents of m ore im portant applications or users.
2 .5 Modeling event processing applications
Modeling event processing applications is an im portant part of the application's
lifecycle. Figure 2.1 [ Etzion 2010] shows an exam ple of such a m odeling system .

30. 30
An event processing application m odel should specify the following:
The types of events used by the application and sem antic m etadata associated
with them
The producers and consum ers of events, and their relation to the event
processing logic that m akes up the application (In figure 2.1, this is shown as an
event processing network, m ade up of event processing agents, representing the
operations perform ed on events by event processing logic. Producers and consum ers
are linked via channels showing the flow of events.)
Context definitions (Many event processing operations depend on the context in
which the event occurred, as m entioned in Section 2.3.)
Stateful data (This m odels data that is read or written by event processing logic as
it handles events that flow through the event processing network Stateful data can
include the following:
1 . Historical event logs and stores,
2 . State of entities external to the event processing application,
3 . Shared processing state used by the application itself,
4 . Reference data used to validate and enrich events.
Preferences (The preceding item s all m odel functional requirem ents of the
application. In addition, the m odel should record the critical non-functional
requirem ents that we listed in section 2.4.)
More inform ation about the requirem ents for event processing system s can be found
in [ Chandy 2009] and [ Etzion 2010] .
Figure 2 .1 . An event processing m odeling for an event
processing netw ork

31. 31
Chapter 3 : Synergies and relations to
other areas
Event processing system s and infrastructure do not exist in a vacuum , but rather
entail m any interactions with other technologies and com puting paradigm s. When
considering the appropriateness of event processing for a given problem ,
understanding the salient differences between event processing and alternative
technologies is helpful. The best approach to a problem m ay require blending event
processing with supporting or com plem entary technologies. Understanding the
differences and overlapping areas between these technologies and event processing
is particularly useful in these cases.
3 .1 An event-driven theft detection as an exam ple of
synergies and relations to other areas
The developm ent of an event-driven sense and respond infrastructure can have an
enorm ous im pact on responses to everyday events in society.
Consider, for exam ple, the occurrence of a theft of m erchandise in a store. Loss of
m erchandise (known euphem istically as shrinkage in the retail industry1
) accounts
for a significant loss of revenue—typically anywhere from 1 to 8% of gross sales—
and results in increased prices passed along to the public. Envision a scenario in
which a person set on illicitly acquiring som e m erchandise enters a store. A video
facial recognition system scans the person's face and processes the im age to
com pute a set of biom etric m arkers. The set of m arkers are contributed to a
continuous stream that is scanned by a law enforcem ent application in turn linked to
a police record data base.
If a positive identification is m ade, m eaning that the person has a record, an alert
will be generated to subscribers that a potential thief has been spotted in the store.
In the store itself, security personnel are then alerted as they subscribe to the law
enforcem ent m onitoring system , and the store video system is instructed to provide
a stream of data relating to the thief's whereabouts in the store. The detection of an
actual theft m ay be perform ed by hum ans, although technology is currently being
developed to detect gestures that m ight connote the action of theft and thus trigger
a theft alert autom atically2
. The identification and location of the thief in the store
are then sent to law enforcem ent along with geo-spatial inform ation locating the
store. These events are com bined with real tim e locations of available police
resources. The closest police team is alerted to respond to the theft. This scenario
could be developed using a range of different technologies, som e involving event
processing, others independent of it.
3 .2 Synergies and relations of event processing and
analytics
In this section, we position event processing alongside analytics to provide guidance
and a possible direction for opportunities that em erge when these disciplines are
com bined. Researching and developing one of the two fields with the other
technology in m ind will help expand the potential of such opportunities.
1
http: / / retail.about.com / od/ glossary/ g/ shrinkage.htm
2
See for exam ple http: / / en.wikipedia.org/ wiki/ Gesture_recognition and references
therein.

32. 32
When we refer to analytics, we forem ost relate to predictive analytics in which data
m ining and m achine learning techniques are used to sift through data for finding
useful patterns and discovering new insights. The resulting m odels can be deployed
to leverage these insights and m ake predictions. Com m on types of predictive m odels
include classification m odels such as regressions, decision trees and neural nets,
clustering m odels, and association m odels.
We also refer to statistical inform ation of the execution of a system , such as
m easuring key perform ance indicators (KPI s) and business intelligence.
3 .2 .1 Learn and discover event processing patterns w ith
predictive analytics
Discovering EP patterns with predictive analytics is a design-tim e, offline interaction
in which sifting through application data can reveal insightful patterns for event
processing. These patterns m ay benefit the system that includes event processing.
Since event processing patterns are usually specified by hum ans, in som e cases,
im portant patterns m ay be overlooked or current patterns m ay not reach their
potential (in avoiding risks or capitalizing on opportunities).
The purpose of predictive analysis is to discover and learn new patterns that can be
deployed to the event processing system or engine.
In our theft exam ple, learning and discovery tools could be used to correlate loss
events with other activities in the store, such as a shift change or an especially busy
tim e. This discovery could be used to im prove theft detection. As seen in figure 3.1,
the data that predictive analytics is applied to m ay or should include data on raw
(incom ing) events, events produced by event processing (derived events), and
scoring m odel results. The m odels produced m ay turn out to be event processing
rules or patterns.
Figure 3.1 illustrates the interactions between event processing and business
analytics.

33. 33
Figure 3 .1 : Event processing and predictive analytics interactions

34. 34
3 .2 .2 I m plem enting and executing predictive m odels w ith
event processing
The result of techniques that learn and discover patterns provide predictive m odels,
but these m odels are often not directly deployed and executed. Som etim es these
m odels are converted to procedural code or to SQL queries running directly on the
data. At other tim es, the m odels are run offline, com pletely off the operational
process that handles the data on which the m odels are to predict. Event processing
m ay be used to execute som e parts of predictive m odels inline with the operational
process. Event processing could be used to collect, aggregate, and correlate events
to provide the m ore inform ative inform ation required to execute som e predictive
m odels. This is som etim es referred to as calculating the features' values for
executing a m odel.
In our loss prevention exam ple, predictive m odels m ight be integrated into police
dispatch m anagem ent. Crim e predictions would be integrated with geospatial
inform ation to direct police resources to the best locations based on historical crim e
profiles and current events.
3 .2 .3 I ncorporate predictive m odels in event processing
In som e cases predictive m odels do get deployed to and executed by an engine. In
these scenarios, an event processing engine/ system m ay call upon services of that
predictive m odel engine, such as a scoring service, and establish m ore sophisticated
pattern detections and actions. The P-ToPSS approach [ Muthusam y 2010] is one
exam ple of an event processing approach that predicts future event m atches based
on observed event histories and on partial event m atches.
A scoring service analytic m odel m ight be used to m easure the likelihood that a
series of activities by a patron represent a potential theft event. If the score crosses
a certain threshold, the patron could be confronted by staff to reduce the potential
for a loss event.
3 .3 Synergies and relations of event processing and
active rules ( ECA)
A strong connection exists between the event-condition-action (ECA) paradigm and
event processing (EP).
ECA has its origins in active databases. The idea is as follows: If an event occurs that
satisfies a condition, then an action takes place. Events, conditions, and actions can
be m ore or less elaborate, depending on the system . Many system s rely on this
paradigm . We can cite HiPAC [ Dayal 1996] am ong the pioneers of this paradigm .

35. 35
EP applies the three basic ECA concepts. The basic concept behind an event is the
sam e, as are the ones behind conditions and actions. However, EP goes beyond ECA
in the sense that it considers m ore com plex events, conditions, and actions. This
im plies that ECA is included in EP, as illustrated in Figure 3.2.
To illustrate that ECA is fully included in EP,
we m ust consider a case of events that is probably the m ost representative. In ECA,
events were considered as sim ple events, in the m anner that new entries in a
database are typically handled. In EP, the event part tends to be m ore sophisticated.
For instance, it m ay consider m any events in com plex expressions (e.g., in a
tem poral conjunction). Other extensions cam e over tim e—for instance, the inclusion
of events from external sources. The fact that events are m ore com plex leads to an
E* CA paradigm . E* can then be a pattern of events, leading to the Pattern-
Condition-Action (PCA) concept, which recently becam e a com m on term in the event
literature.
In our loss prevention exam ple, ECA rules are sufficient for som e processing, such as
alerting law enforcem ent when a theft occurs. Yet ECA rules are insufficient for m ore
com plex conditions and actions, such as identifying the best available police
resource.
3 .4 Synergies and relations of event processing and
publish-subscribe approach
The publish-subscribe approach is concerned with the com m unication
(dissem ination) of events. The approach [ Eugster 2003, Fabret 2001] is em erging as
an appropriate com m unication paradigm for large-scale system s. It allows loose
coupling between m utually anonym ous com ponents and supports m any-to-m any
com m unication. In the publish/ subscribe paradigm , a principal takes the role of a
publisher and/ or a subscriber. Principals connect to the publish/ subscribe m iddleware
in order to com m unicate. Publishers advertise the events they are prepared to
publish. Subscribers register their interest in receiving events through a subscription
that the m iddleware handles. Publishers produce events without any dependence on
subscribers.
Figure 3 .2 . Relation betw een event processing and business rules ( ECA)

36. 36
This process occurs through an event broker, which routes—typically in cooperation
with other brokers—events from publishers to subscribers. An event is delivered to a
subscriber if it m atches a subscription. This process is term ed notification [ Eugster
2003, Fabret 2001] .
Publish-subscribe system s are available as type/ topic or content/ attribute-based.
Topic-based publish-subscribe system s involve the association of an event channel
with a particular nam ed topic/ type. Producers publish events to the appropriate
channel, while subscribers express their interest in receiving m essages of a certain
type. Content-based publish-subscribe system s consider m essage content—a
subscriber defines his interest in receiving particular events based on the type and
attribute values of the event.
3 .4 .1 I ntegrating publish-subscribe w ith event processing
The publish-subscribe approach is defined in term s of com m unicating individual
events that are published and m atch subscriptions. The detection of patterns
involving a num ber of different event types, also known as com plex event
processing, is typically built as one or m ore services above the dissem ination
network or is directly integrated into the publish-subscribe substrate as in the
PADRES system [ Fidler 2005, Li 2008, and Jacobsen 2009] .
These services use publish-subscribe m iddleware. At the lowest level, they m ay
subscribe to what is referred to as prim itive events, such as sensor readings, and
having detected a defined pattern, publish a higher-level com posite event. In
general, an event processing service m ay subscribe to a com bination of prim itive and
com posite events. The EP services m ay be deployed to distribute sub-trees of a
required pattern throughout a network to m inim ize com m unication overhead. An
exception to the layered m odel just described is the PADRES system [ Jacobsen
2009] , in which event processing is integrated within the broker network.
In our theft exam ple, a publish-subscribe system m ight be used by the police
departm ent to distribute alerts about crim inal activity, or by m erchants to distribute
reports am ongst their peers. Without a publish-subscribe system , creating an event
processing system that crosses adm inistrative and organizational boundaries would
be difficult.
Figure 3 .3 . Relation betw een event processing and the publish- subscribe approach

37. 37
3 .5 Synergies and relations of event processing and
business process m anagem ent ( BPM)
Business process m anagem ent is a broad field that covers several areas, including
the capturing, analysis, and design of business processes (such as the order-to-cash
process). The result of these activities is often referred to as a business process
m odel or business process type. Business process m anagem ent also incorporates the
autom atic execution of processes according to these business process m odels,
resulting in business process instances (e.g., fulfillm ent of a specific order). Process
steps, as defined in the process m odel, can either be autom ated activities (e.g.,
calling an appropriate service) or hum an tasks. The steps are scheduled according to
a definition of the work flow in the business process m odel. Finally, business process
m anagem ent also com prises the m onitoring of business process instances, the
capturing and analyzing process and step duration, and the waiting tim es, etc. From
these m onitoring activities, often referred to as business activity m onitoring (BAM),
bottlenecks and optim ization potential can be identified.
3 .5 .1 I m plem enting BPM w ith event processing
Business process m anagem ent can benefit from event processing in several areas.
First, event processing can be used to im plem ent the execution of business process
instances [ Li 2010, Muthusam y 2009] . Each step can be seen as a separate unit of
execution that when finished em its an event that is then captured by an event
processing engine. This in turn starts the next activity, based on the business
process m odels. Business process m odels typically include rules to determ ine the
sequence of steps that have to be evaluated by the event processing engine
(com plex event processing) used to determ ine which step is to be executed next
(orchestration, choreography). Analogously, business processes can be
hierarchically-layered, i.e., a process execution triggers the execution of a
subordinate process.
3 .5 .2 Monitoring processes w ith event processing
Process execution m onitoring is another area in which event processing is beneficial.
Events can occur on the m eta level (in which step x has been started) or on the data
level (e.g., order received for part 124). Com plex event processing is used to identify
trends (e.g., execution tim e is growing in the afternoon) and critical situations and to
throw alarm s (e.g., critical order not processed within 24 hours), etc. The results can
be used for real-tim e reporting and dashboards. The tim e awareness and correlation
capabilities of com plex event processing add functionality and flexibility to BAM.
3 .5 .3 I nfluencing business processes w ith event processing
In a m ore advanced scenario, the execution of business process instances can be
influenced by event processing beyond the definitions in the process m odel. For
exam ple, event processing can initiate the execution of an additional activity steps in
a process (e.g., due to recently detected fraud activities, all currently processed
orders exceeding $100 m ust be reviewed by a m anager). In addition, business
process instances could be initiated by event processing (e.g., order m aterial
because events indicated upcom ing stock shortage) or even stopped (e.g., cancel all
orders from a fraudulent custom er). In cases like these, com plex event processing
allows for flexible and tim ely reactions in threat or opportunity situations.
In sum m ary, business process m anagem ent can benefit from event processing,
because event processing allows for continuous tim ely reactions and responsiveness.

38. 38
Event processing adds agility, flexibility, and adaptivity to BPM, by opening up the
back box of a business process. Moreover, the loose coupling that is im plicit in event
processing enables adding these feature in a non-intrusive m anner.
3 .6 Synergies and relationships of event processing
and data stream s
This section discusses the relationship between the notion of event processing and
the notion of stream s, an issue that is a source of frequent confusion.
3 .6 .1 Brief overview of stream s
A data stream m anagem ent system (also called a stream ing system ) is a platform
for developing and deploying stream ing applications that run continuous queries
(CQs) over incom ing stream s. A stream is a potentially unbounded sequence of
events (also called tuples), usually arranged in som e order. Stream ing system s
typically consider an event to be a notification that contains a payload with a well-
defined schem a sim ilar to table schem as in traditional databases. In som e stream ing
system s, events also have a control field, providing m etadata, such as tem poral or
sequencing inform ation. Stream ing system s are typically characterized by their well-
defined query sem antics [ Barga 2007, Soule 2010] and som etim es have a
declarative language [ Jain 2008, LINQ] . At the execution level, queries in stream ing
system s consist typically of a directed graph of operators (sim ilar to relational
operators in database system s) that process events using a dataflow-style event
processing paradigm . Com m on operators include selection, projection, join,
aggregation, and grouping. A data stream is just one type of stream , and the term
stream has wider coverage that includes video stream s and voice stream s, etc.
Figure 3 .4 . Relation betw een event processing and business process
m anagem ent ( BPM)