SlideShare a Scribd company logo
epts
  event processing technical society




                                                                 Tutorial v1.2




                                       epts
                                       event processing technical society
                                       Content by: members of the
                                       EPTS Reference Architecture Working Group



Event Processing                            Adrian Paschke
                                              (Freie Universitaet Berlin)
                                            Paul Vincent
Reference Architecture                        (TIBCO Software)
                                            Catherine Moxey
and Design Patterns                           (IBM)
                                            Alex Alves
                                              (Oracle)
epts
    event processing technical society




Tutorial

       • Event Processing is an increasingly “mainstream”
            view in IT, if not widely in production
       • Event Processing Functional Reference Architecture
            (FRA) describes the functions involved in event
            processing
       • Functions in the FRA can themselves be described as
            event processing patterns
       • Most technology providers can support the FRA and
            its associated design patterns


2
epts
    event processing technical society




Agenda

       • Introduction to the EPTS Functional Reference
            Architecture, and what it contains
       • Mapping Functions to Patterns: some sample
            Event Processing Patterns
       • EPTS member pattern implementation examples
       • Summary and future work on EP Patterns




3
epts
    event processing technical society




Introduction to Event Processing

       • Event-centric view of IT
       • Events are
            – Sent and Received
            – Aggregated, Transformed into Data, Deleted
            – Processed in queries, rules etc
            – Cause actions like processes, service invocations, etc




4
epts
          event processing technical society



Introduction to Event Processing

                                                                  Event Processing,
                                                              Complex Event Processing,
                                                               Event Stream Processing
                           Event Processing Rules, etc.
 Event Type Definitions,




                                                             Event Abstraction, Event Pattern
                                                              Detection, Event Composition




                                                                                                            Event Management
                                                                           etc. etc.



                                                                Events                 Derived
                                                                                       Events



                                                          Event Producer           Event Consumer
                                                          (Event Source,           (Event sink, event
                                                          Event Emitter)             handler, event
                                                                                       listener)

Design time                                                                Run time                     Administration
epts
  event processing technical society



Motivation and Benefits

• Motivation
  – Event Processing is evolving and exploiting many technologies
  – Potential adopters (stakeholders) need a reference to
    understand suppliers‟ architectures and solutions.
• Benefits
  – a Reference Architecture predefines
    customizable abstract frames of reference
    for specific stakeholder concerns and application domains.
     • Aids reuse of successful EP architectures
       for frequently occurring EP design problems
  – Underlying Architecture is a Reference Model that defines the
    terminology and components in Event Processing architectures
epts
      event processing technical society




Reference Architecture Viewpoints
     Viewpoint                                                   Viewpoint
     Element                               Engineering EP   Managing EP         Business with EP
                                           Architecture     Architecture        Architecture
     Concepts                              How to           How to apply?       How to utilize / sell /
                                           implement?                           own?
     Stakeholders Architects /                              Project Manager Decision Maker,
                  Engineers                                                 Customer, Provider
     Concerns                              Effective        Operational         Strategic and
                                           construction and Management          tactical
                                           deployment                           management
     Techniques /                          Modeling,        IT (service/appl)   Monitoring,
     Languages                             Engineering      management,         Enterprise Decision
                                                            project             Management,
                                                            management          Governance
10
epts
      event processing technical society




Stakeholders and Viewpoints covered

                                                    Viewpoints (for CEP / stateful EP)
     Stakeholder                           Component                  Functional
     Decision Maker                        [Not applicable]           Inputs, Outputs and
     / End User                                                       Processing Requirements
                                                                      View
     Architect /                           Solution Components        Functions carried out in
     Designer                              View                       CEP View



                                                 TBD (2012?)                Focus so far


11
epts
  event processing technical society




Functional Reference Architecture

     • Architect and Developer perspective
          – includes the 3 main functions
            (development, run-time and administration),
          – targets primarily the automated event processing
            operations
     • Run-time functions in 2 main groups:
          – the event infrastructure (sources and consumers)
            external to the event processor under consideration,
          – the event processor.
epts
          event processing technical society




Functional View source: definitions of EP
                                                              Event Processing, Complex
                                                               Event Processing, Event
                                                                 Stream Processing
                           Event Processing Rules, etc.
 Event Type Definitions,




                                                             Event Abstraction, Event Pattern
                                                              Detection, Event Composition




                                                                                                            Event Management
                                                                           etc. etc.



                                                                Events                 Derived
                                                                                       Events



                                                          Event Producer           Event Consumer
                                                          (Event Source,           (Event sink, event
                                                          Event Emitter)             handler, event
                                                                                       listener)

Design time                                                                Run time                     Administration
epts
              event processing technical society




Reference Architecture: Functional View
 Definition, Modeling, (continuous) Improvement


                                                             Event Reaction
                                                      Assessment, Routing, Prediction,
                                                            Discovery, Learning
       (Pattern, Control, Rule, Query, RegEx.etc)




                                                                                                                       Event Process Monitoring, Control
                                                                                         0..*




                                                                                                State Management
                                                        Complex Event Detection
              Event and Complex Event




                                                        Consolidation, Composition,
                                                                Aggregation
                                                                                         0..*
                                                              Event Analysis
                                                      Analytics, Transforms, Tracking,
                                                       Scoring, Rating, Classification
                                                                                         0..*
                                                             Event Preparation
                                                     Identification, Selection, Filtering,
                                                          Monitoring, Enrichment
                                                                                         0..*


                                                    Event Production       Event Consumption
                                                       Publication,         Dashboard, Apps,
                                                        Retrieval            External Reaction


Design time                                                         Run time                                       Administration
epts
  event processing technical society




Reference Architecture: Functional View / Runtime

                                              Event Reaction
                                       Assessment, Routing, Prediction,
                                             Discovery, Learning
                                                                           0..*




                                                                                  State Management
                                          Complex Event Detection
                                          Consolidation, Composition,
                                                  Aggregation
                                                                           0..*
                                                Event Analysis
                                        Analytics, Transforms, Tracking,
                                         Scoring, Rating, Classification
                                                                           0..*
                                               Event Preparation
                                       Identification, Selection, Filtering,
                                            Monitoring, Enrichment
                                                                           0..*


                                   Event Production          Event Consumption
                                      Publication,            Dashboard, Apps,
                                       Retrieval               External Reaction


                                                      Run time
epts
  event processing technical society




Reference Architecture: Functional View / Runtime


Event Production: the source of events for event processing.
• Event Publication: As a part of event production, events may be published
  onto a communication mechanism (eg event bus) for use by event
  consumers (including participants in event processing). This is analogous to
  a "push" system for obtaining events.
• Event Retrieval: As a part of event production, events may be explicitly
  retrieved from some detection system. This is analogous to a "pull" system
  for obtaining events.



                                   Event Production   Event Consumption
                                      Publication,     Dashboard, Apps,
                                       Retrieval        External Reaction
epts
  event processing technical society




Reference Architecture: Functional View / Runtime

Event Consumption: the process of using events from event publication and
processing. Event processing itself can be an event consumer, although for
the purposes of the reference architecture, event consumers are meant to
indicate downstream consumers of events generated in event processing.
• Dashboard: a type of event consumer that displays events as they occur to
  some user community.
• Applications: a type of event consumer if it consumes events for its own
  processes.
• External Reaction: caused through some event consumption, as the result
  of some hardware or software process.

                                   Event Production   Event Consumption
                                      Publication,     Dashboard, Apps,
                                       Retrieval        External Reaction
epts
    event processing technical society




Reference Architecture: Functional View / Runtime
Event Preparation: the process of preparing the event and associated payload and
metadata for further stages of event processing.
• Entity Identification: incoming events will need to be identified relative to prior events, such
  as associating events with particular sources or sensors.
• Event Selection: particular events may be selected for further analysis. Different parts of
  event processing may require different selections of events. See also event filtering.
• Event Filtering: a stream or list of events may be filtered on some payload or metadata
  information such that some subset is selected for further processing.
• Event Monitoring: particular types of events may be monitored for selection for further
  processing. This may utilise specific mechanisms external to the event processing such as
  exploiting event production features.
• Event Enrichment: events may be "enriched" through knowledge gained through previous
  events or data.



          Event Preparation
  Identification, Selection, Filtering,
       Monitoring, Enrichment
epts
   event processing technical society




Reference Architecture: Functional View / Runtime
 Event Analysis: the process of analysing suitably prepared events and their
 payloads and metadata for useful information.
 • Event Analytics: the use of statistical methods to derive additional information about an
   event or set of events.
 • Event Transforms: processes carried out on event payloads or data, either related to
   event preparation, analysis or processing.
 • Event Tracking: where events related to some entity are used to identify state changes in
   that entity.
 • Event Scoring: the process by which events are ranked using a score, usually as a part of
   a statistical analysis of a set of events. See also Event Analytics
 • Event Rating: where events are compared to others to associate some importance or
   other, possibly relative, measurement to the event.
 • Event Classification: where events are associated with some classification scheme for
   use in downstream processing.

          Event Analysis
  Analytics, Transforms, Tracking,
   Scoring, Rating, Classification
epts
   event processing technical society




Reference Architecture: Functional View / Runtime

Complex Event Detection: the process by which event analysis results in the
creation of new event information, or the update of existing complex events.
• Event Consolidation: combining disparate events together into a "main" or "primary"
  event. See also event aggregation.
• Event Composition: composing new, complex events from existing, possibly source,
  events.
• Event Aggregation: combining events to provide new or useful information, such as trend
  information and event statistics. Similar to event consolidation.




    Complex Event Detection
    Consolidation, Composition,
            Aggregation
epts
   event processing technical society




Reference Architecture: Functional View / Runtime
Event Reaction: the process subsequent to event analysis and complex event
detection to handle the results of analysis and detection.
• Event Assessment: the process by which an event is assessed for inclusion in some
  process, incorporation in some other event, etc.
• Event Routing: the process by which an event is redirected to some process, computation
  element, or other event sink.
• Event Prediction: where the reaction to some event processing is that some new event is
  predicted to occur.
• Event Discovery: where the reaction to some event processing is the disclosure of a new,
  typically complex, event type.
      • Note that event prediction is predicting some future event, usually of a known type, whereas event
        discovery is the uncovering of a new event type. See also event-based learning.

• Event-based Learning: the reaction to some event processing that uses new event
  information to add to some, typically statistical-based, understanding of events.
                                                • Note that event-based learning is a specialisation of
                                                  general machine learning and predictive analytics.
        Event Reaction
 Assessment, Routing, Prediction,
       Discovery, Learning
epts
              event processing technical society




Reference Architecture: Functional View / Design time
 Definition, Modeling, (continuous) Improvement




                                                    Covers the definition, modeling, improvement /
       (Pattern, Control, Rule, Query, RegEx.etc)




                                                    maintenance of the artifacts
                                                    used in event processing:
              Event and Complex Event




                                                    • event definitions, including event metadata and payloads,
                                                    • event and event object organisations and structures,
                                                    • event processing transformations / queries / rules /
                                                      procedures / flows / states / decisions /
                                                      expressions (although these can sometimes be
                                                      considered as administrative updates in some situations)




Design time
epts
  event processing technical society




Reference Architecture: Functional View / Administration

   Administrative concepts of




                                                                      Event Process Monitoring, Control
   monitoring and control. This may involve
   • starting and stopping the application and event
     processing elements, including application monitors
   • providing and updating security levels to event inputs and
     outputs (also can design-time)
   • management of high availability and reliability resources,
     such as hot standby processes
   • resource utilisation monitoring of the event processing
     components
   • process updates, such as how-swapping of event
     processing definitions to newer versions.

                                                                  Administration
epts
     event processing technical society




Agenda

        • Introduction to the EPTS Functional Reference
             Architecture, and what it contains
        • Mapping Functions to Patterns: some sample
             Event Processing Patterns
        • EPTS member pattern implementation examples
        • Summary and future work on EP Patterns




24
epts
     event processing technical society




Event Processing Patterns

        • Functions from Reference Architecture are a
             guide to possible low-level patterns
                                                 Event Reaction
                                          Assessment, Routing, Prediction,
                                                Discovery, Learning

                                             Complex Event Detection
                                             Consolidation, Composition,
                                                     Aggregation

                                                   Event Analysis
                                           Analytics, Transforms, Tracking,
                                            Scoring, Rating, Classification

                                                  Event Preparation
                                          Identification, Selection, Filtering,
                                               Monitoring, Enrichment

25
epts
     event processing technical society




Which Classification of Patterns to Use?

        • Common EP Functions?
             As described in the Reference Architecture
        • Other sources?
             Recent literature




26
epts
       event processing technical society



Comparison of EPTS RA Fns and EPIA EP Agents
        Event Reaction
 Assessment, Routing, Prediction,
       Discovery, Learning

      Complex Event Detection
      Consolidation, Composition,
              Aggregation

             Event Analysis
     Analytics, Transforms, Tracking,                                                  EPA
      Scoring, Rating, Classification

        Event Preparation                                         Filter         Transformation       Pattern detect
Identification, Selection, Filtering,
     Monitoring, Enrichment

                                                     Translate             Aggregate          Split              Compose




                                            Enrich               Project


27
epts
     event processing technical society



Comparison of EPTS RA Fns and EPIA EP Agents
Event Preparation




28
epts
     event processing technical society



Comparison of EPTS RA Fns and EPIA EP Agents
Event Analysis




29
epts
     event processing technical society



Comparison of EPTS RA Fns and EPIA EP Agents
Complex Event Detection




30
epts
     event processing technical society



Comparison of EPTS RA Fns and EPIA EP Agents
Event Reaction




31
epts
     event processing technical society




What do we want to cover in an EP Pattern?



              Type /                      Operands          Operation /
             synonym                                        Operators

       Event Filtering: a stream or list of events may be filtered
              on some payload or metadata information
       such that some subset is selected for further processing.

                                                      Rationale /
               Mapping to                            business rule
              EPL template

32
epts
     event processing technical society




EP MetaPattern version 1

        • Name & Alternative Names
        • Classification versus other references
             – EPTS RA and other references e.g. “EP in Action” book
        • Description
             – Role, and Business Rule type specification / requirement
        • Structure
             – EPTS Glossary terms
        • Implementations

33
epts
     event processing technical society




EP MetaPattern future versions?

        • Name & Alternative Names
        • Classification versus other references
             – EPTS RA and other references e.g. “EP in Action” book
        • Description                         may use revised EP Fns
                                              classification / ontology
             – Role, and Business Rule type specification / requirement
        • Structure
             – EPTS Glossary terms
        • Implementations                   may use EPL classification
                                            per EPTS Languages WG
34
epts
     event processing technical society




EP Pattern Implementations
        • Differ in Implementation type (Event Processing
             Language, Execution semantics, etc)
        • Sampled systems we cover:
        1.TIBCO BusinessEvents Rete-production-rule-engine
             (with continuous query and event-expression options)
        2.Oracle CEP Stream-processing engine
        3.Prova Logic Programming Semantic CEP Rule Engine
        4.IBM WebSphere Decision Server event-driven
             production-rule-engine
             & IBM Infosphere Streams stream processing engine
35
epts
     event processing technical society




Agenda

        • Introduction to the EPTS Functional Reference
             Architecture, and what it contains
        • Mapping Functions to Patterns: some sample
             Event Processing Patterns
        • EPTS member pattern implementation examples
        • Summary and future work on EP Patterns




36
epts
     event processing technical society




Some EP Patterns to review in more detail
        1.        Filter: filter out all events that have some property in their
                  payload / data
             –         E.g. customer purchases: filter out those with values < $100
        2.        Enrichment: add some information based on prior events
             –         E.g. customer purchase: add the customer history to the
                       purchase event
        3.        Aggregation: add prior events together for new information
             –         E.g. customer purchase history: track the sum total of order
                       values for a customer
        4.        Routing: based on the purchase type pass the event on to the
                  appropriate service
             –         E.g. customer purchase event: pass on to a provisioning service
37
                       based on the type of product / product classification
epts
     event processing technical society




Some EP Patterns to review in more detail
        1.        Filter: filter out all events that have some property in their
                  payload / data
             –         E.g. customer purchases: filter out those with values < $100
        2.        Enrichment: add some information based on prior events
             –         E.g. customer purchase: add the customer history to the
                       purchase event
        3.        Aggregation: add some information based on prior events
             –         E.g. customer purchase history: track the sum total of order
                       values for a customer
        4.        Routing: based on the purchase type pass the event on to
                  the appropriate service
             –         E.g. customer purchase event: pass on to a provisioning
38
                       service based on the type of product / product classification
epts
     event processing technical society




Filter Pattern: Classification (1)

        • EPTS Reference Architecture:
             – "During event preparation, a stream or list of events may
               be filtered on some payload or metadata information
               such that some subset is selected for further processing."
        • EPIA:
             – Filter (EPA or Event Processing Agent) performs filtering
               only and has no matching or derivation steps, so it does
               not transform the input event.




39
epts
     event processing technical society




Filter Pattern: Description (2)

        • Role / Explanation
             – Comparing some property of the event (an attribute or
               metadata) with some other value (from some other event, or
               data)
        • Associated Business Rule Specification
             – As a constraint: a selection (to be enforced during the
               processing of events):
               All <event entities>
               that have                                 Filter
               <filter expression>
                                          event                        event
                                                       f(event)
               must have
               <some status>.                            discarded
                                                        event
40
epts
     event processing technical society




Filter Pattern: Structure (3)

        • EPTS Glossary comparison
             – A filter pattern can relate to several terms in the EPTS
               Glossary
                  • can be specified in an event pattern (A template containing
                    event templates, relational operators and variables...)
                  • .. by an event pattern constraint (A Boolean condition that
                    must be satisfied by the events observed in a system...)
                  • .. as part of an event processing rule (A prescribed method
                    for processing events.)
                  • .. or as part of event stream processing (Computing on inputs
                    that are event streams.)
             –
41
epts
     event processing technical society




Filter Pattern: Implementations (4)

        • TIBCO, Oracle, Prova and IBM implementations for
             this pattern




                                                    Filter
                                          event                event
                                                  f(event)

                                                   discarded
                                                     event


42
epts
     event processing technical society



Filter Pattern: Implementation 1a
TIBCO BusinessEvents: rules
        • General pattern:
             <declare> event                           event

             <if> event-boolean-expression              Filter
                                                      f(event)

             <then> update-event-with-fact             event
             OR
             action-for-filtered-event
                                                    Operation on
        • Effect: ignore filtered events               event
             on operations on non-filtered events


43
epts
     event processing technical society



Filter Pattern: Implementation 1b
TIBCO BusinessEvents: event removal rules
        • Implementation type: Rete-based rule-driven system
        • Alternate pattern:
             <declare> event
                                                         event
             <if>
                                                       NOT Filter
             NOT event-boolean-expression
                                                        !f(event)
             <then>
             consume-event                               event

        • Effect: removed filtered events from
             system (and ALL subsequent operations!)

44
epts
     event processing technical society



Filter Pattern: Implementation 1c
TIBCO BusinessEvents: State transitions
        • General pattern:
             <state models for concept C>             event
             <route from State n to n+1>
             <on> event                               Filter
             <where> event-filter                  query(event)

        • Effect: change state for some concept       State
                                                    transition
        • Notes: this is a specialisation of the
             general rule pattern!


45
epts
     event processing technical society



Filter Pattern: Implementation 1d
TIBCO BusinessEvents: Queries
        • General pattern:
             <select> output                          event
             <from> event
             <where> event-filter                    Filter
                                                  query(event)

             <policy> notification               Policy   Event
                                                 event    subset
        • Effect: collect filtered events for
             event operations per some policy
                                                 Operations on
                                                  event subset
        • Note: the pattern matching framework   on policy event
             has a similar notion
46           but a different surface syntax
epts
     event processing technical society



Filter Pattern: Implementation 2a
Oracle CEP: queries

       • General pattern:                            event
            SELECT <property>
            FROM <STREAM>                          predicate
            WHERE <predicate-condition>

                                                     event


       • Effect: discard events whose predicate   Operation on
            evaluate to false                        event


47
epts
     event processing technical society



Filter Pattern: Implementation 2b
Oracle CEP: model

        • Scenario:
             – Select only stocks from the stock tick stream whose
               symbols are „AAA‟.
        • Application Model




48
epts
     event processing technical society



Filter Pattern: Implementation 2c
Oracle CEP: query

                                          Specify STREAM source


             SELECT *
             FROM StockTickStream
                                                   Define predicate for
             WHERE symbol = „AAA‟                        filtering




49
epts
   event processing technical society



Filter Pattern: Implementation 3a
Prova: rules
• Implementation type: backward reasoning logical filter
• General pattern template:
 rcvMsg / rcvMult <Event Msg Pattern(s)> :- event
      <logical filter condition(s)>,
                                                  … event
      ... .
                                                        Event pattern +
 <logical derivation rule 1> :-                         logical rule filter
      <filter conditions>.
                                            Backward
  <logical derivation rule 2> :-            Resoning
                                              Rule
                                                              event
       <filter conditions>.                  chaning
      ...

• Effect: Unify received events with                      Operation on
   event pattern definition and apply logical                event
50 filter rules (derivation rules with rule chaining)
epts
     event processing technical society



Filter Pattern: Implementation 3b
Prova: Messaging Reaction Rules in Prova
•   Send a message
sendMsg(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)
•   Receive a message
rcvMsg(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)
•   Receive multiple messages
rcvMult(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)


Description:
 – XID is the conversation identifier
    – Protocol: protocol e.g. self, jms, esb etc.
    – Agent: denotes the target or sender of the message
    – Performative: pragmatic context, e.g. FIPA Agent Communication
    – [Predicate|Args] or Predicate(Arg1,..,Argn): Message payload
epts
  event processing technical society

Filter Pattern: Implementation 3c
Prova: Example
 % Filter for stocks starting with „A“ and price > 100
 rcvMult(SID,stream,“S&P500“, inform, tick(S,P,T)) :-
    S = "A.*",
    P > 100,
    sendMsg(SID2,esb,"epa1", inform, happens(tick(S,P),T).

Example with rule chaining
   % Filter for stocks starting with „A“ and price > 100
   rcvMult(SID,stream,“S&P500“, inform, tick(S,P,T)) :-
        filter(S,P),
        sendMsg(SID2,esb,"epa1", inform, happens(tick(S,P),T).
   filter(Symbol,Price) :-
                           Price > 100,
                           Symbol = "A.*".
epts
    event processing technical society

  Filter Pattern: Implementation 3d
  Prova: Semantic Event Processing Example
  Semantic Query Filer:
  Stocks of companies, which have production facilities in Europe and
  produce products out of metal and
  Have more than 10,000 employees.
                Event Stream
                                              {(Name, “OPEL”)(Price, 45)(Volume, 2000)(Time, 1) }
                                         {(Name, “SAP”)(Price, 65)(Volume, 1000) (Time, 2)}

                        {(OPEL, is_a, car_manufacturer),
                         (car_manufacturer, build, Cars),
                          (Cars, are_build_from, Metall),
   Semantic        (OPEL, hat_production_facilities_in, Germany),
Knowledge Base               (Germany, is_in, Europe)
                        (OPEL, is_a, Major_corporation),
               (Major_corporation, have, over_10,000_employees)}

  rcvMult(SID,stream,“S&P500“, inform,
    tick(Name^^car:Major_corporation,P^^currency:Dollar,
    T^^time:Timepoint)) :-   … <semantic filter inference> .
epts
      event processing technical society



     Filter Pattern: Implementation 4a
     IBM WebSphere Decision Server: Business Events
     component
                                                       event   event
     • Example:
       Define a filter to only process events with a
                                                          Filter
       purchase value greater than £100

                                                          event


                                                       Operation on
      • Effect: Only process events
                                                          event
        which match the filter


55
epts
     event processing technical society




Filter Pattern: Implementation 4b
IBM InfoSphere Streams: Streams Processing Language
 • Example:
 composite Main {
   graph
     stream<rstring name, uint32 age> Beat = Beacon() {}
     stream<rstring name, uint32 age> Youngs = Filter(Beat)
     {
       param filter : age < 30u;
     }
     (stream<Beat> Younger; stream<Beat> Older) = Filter(Beat)
     {
       param filter : age < 30u;
     }
 }

     • Effect: filter out into streams of older and younger ages
     56
epts
     event processing technical society




Some EP Patterns to review in more detail
        1.        Filter: filter out all events that have some property in their
                  payload / data
             –         E.g. customer purchases: filter out those with values < $100
        2.        Enrichment: add some information based on prior events
             –         E.g. customer purchase: add the customer history to the
                       purchase event
        3.        Aggregation: add some information based on prior events
             –         E.g. customer purchase history: track the sum total of order
                       values for a customer
        4.        Routing: based on the purchase type pass the event on to
                  the appropriate service
             –         E.g. customer purchase event: pass on to a provisioning
57
                       service based on the type of product / product classification
epts
     event processing technical society




Enrichment Pattern: Classification (1)

        • Alternative Names:
             – Event Annotation, Semantic Event
        • EPTS Reference Architecture:
             – "During event preparation, events may be "enriched"
               through knowledge gained through previous events or
               data. "
        • EPIA:
             – Enrich (EPA): a translate EPA that takes a single input
               event, uses it to query data from a global state element,
               and creates a derived event which includes the attributes
               from the original event, possibly with modified values...
58
epts
     event processing technical society




Enrichment Pattern: Description (2)

        • Role / Explanation
             – Updating some attribute or metadata of the event with some
               other values, possibly computed, from some other event or
               data
        • Associated Business Rule Specification
             – a selection (to be enforced during the processing of events):
               All <event entities> that are also <enriched data
               expression>
                                                        Other data
               must have <some status>.                  or event


                                                   Enrichment
                                          event                       event
                                                    f(event)
59
epts
     event processing technical society




Enrichment Pattern: Structure (3)

        • EPTS Glossary comparison
             – Enrichment can be specified in an event pattern (pattern
               definition containing additional meta data, semantic
               knowledge)
             – .. as part of an event processing rule (A method for
               enriching events during processing, e.g., by analysing the
               meta data and querying external data sources or
               inferencing semantic knowledge bases)
             – .. or as part of event stream processing (additional
               additional information to event stream data, e.g. by
               querying external data sources)

60
epts
     event processing technical society




Enrichment Pattern: Implementations (4)

        • TIBCO, Oracle, Prova and IBM implementations for
             this pattern



                                                    Other data
                                                     or event


                                                  Enrichment
                                          event                  event
                                                   f(event)



61
epts
     event processing technical society



Enrichment Pattern: Implementation 1
TIBCO BusinessEvents: Rules
        • General pattern:                    Enrich
             <declare> event, data             data
                                                          event

             <if> event-data-match                Identify
                                              enrichment data

             <then> update-event-with-data        Enrich
                                                 Operation
        • Effect: find appropriate data for      on event
             events and apply enrichment
             operations on them
                                                       Event


62
epts
     event processing technical society



Enrichment Pattern: Implementation 2a
Oracle CEP: queries
        • General pattern template:
                                                      event
            SELECT <property>
            FROM <STREAM>[WIN],<RELATION>
                                                  join-condition
            WHERE <join-condition>

                                                      event
        • Effect: enrich events from stream
             with contextual data from relation
             where condition applies.             Operation on
                                                     event


63
epts
     event processing technical society



Enrichment Pattern: Implementation 2b
Oracle CEP: model

        • Scenario:
             – For every stock event from stock tick stream whose bid is
               greater than 5.0, find its company‟s location.
        • Application Model




64
epts
     event processing technical society



Enrichment Pattern: Implementation 2c
Oracle CEP: queries


                                             No need for window
             SELECT event.symbol, location  operator on a „relation‟
             FROM
              StockTickStream [NOW] AS event,
              StockTable AS data
             WHERE event.symbol = data.symbol AND
              event.lastBid > 5.0




65
epts
     event processing technical society



Enrichment Pattern: Implementation 2d
Oracle CEP: table metaphor
               StockTable
               Symbol                              Full-name          Location
               AAA                                 The AAA company    San Francisco

               BBB                                 The BBB company    San Jose



               Time                 Input (StockTickStream)     Output (QUERY)
               1                          {“AAA”, 10.0, 10.5}         {“AAA”, “San…”}

               2                          {“BBB”, 11.0, 12.5}        {“BBB”, “San Jose”}

               3                           {“AAA”, 4.0, 4.5}


66
epts
      event processing technical society



Enrichment Pattern: Implementation 3a
Prova: rules
• General pattern template:
     rcvMsg / rcvMult <Event Msg Pattern(s)> :-   event
          <query built-in(s)>,
          <enrichment condition(s)>                     External
          ... .                               Query      data
     <logical enrichment rule 1> :-
          <enrichment operations>.               Enrichment
                                                    rules

• Effect: query data from external source           event*
     and enrich the event
                                                 Operation on
                                                enriched event
67
epts
     event processing technical society

Enrichment Pattern: Implementation 3b
Prova: External Data and Object Integration
•   Java
    ..., L=java.util.ArrayList(),, ..., java.lang.String(“Hello”)

•   File Input / Output
    ..., fopen(File,Reader), ...

•   XML (DOM)
    document(DomTree,DocumentReader) :-
         XML(DocumenReader),
•   SQL
    …,sql_select(DB,cla,[pdb_id,“1alx“],[px,Domain]).
•   RDF
    ...,rdf(http://...,"rdfs",Subject,"rdf_type","gene1_Gene"),
•   XQuery
    ..., XQuery = ' for $name in
    StatisticsURL//Author[0]/@name/text() return $name',
    xquery_select(XQuery,name(ExpertName)),
•   SPARQL
    ...,sparql_select(SparqlQuery,name(Name),class(Class),
    definition(Def)),
epts
   event processing technical society

Enrichment Pattern: Implementation 3c
Prova: Example with SPARQL Query
% Filter for car manufacturer stocks and enrich the stock tick
  event with data from Wikipedia (DBPedia) about the manufacturer
  and the luxury cars
rcvMult(SID,stream,“S&P500“, inform, tick(S,P,T)) :-
  carManufacturer(S,Man),   % filter car manufacturers
  findall([luxuryCar|Data],luxuryCar(Man,Name,Car),Data), % query
  EnrichedData = [S,Data], % enrich with additional data
  sendMsg(SID2,esb,"epa1", inform, happens(tick(EnrichedData,P),T).

% rule implementing the query on DBPedia using SPARQL query
luxuryCar(Manufacturer,Name,Car) :-
  Query="SELECT ?manufacturer ?name ?car   % SPARQL RDF Query
       WHERE {?car <http://purl.org/dc/terms/subject>
       <http://dbpedia.org/resource/Category:Luxury_vehicles> .
              ?car foaf:name ?name .
              ?car dbo:manufacturer ?man .
              ?man foaf:name ?manufacturer.
       } ORDER by ?manufacturer ?name“,
sparql_select(Query,manufacturer(Manufacturer),name(Name),car(Car)).
epts
     event processing technical society


 Enrichment Pattern: Implementation 4
 IBM WebSphere Decision Server: Business Events
 component
     • Example: Define a Mapped Key expression, where a
       field in the event is used to look up objects in a
       database table
                                                               event

                                                            Mapped Key
                                                              look up

                                                             Enriched
                                                              event

          • Effect: required enrichment data is obtained
70
            from the data base and added to event
epts
     event processing technical society




Some EP Patterns to review in more detail
        1.        Filter: filter out all events that have some property in their
                  payload / data
             –         E.g. customer purchases: filter out those with values < $100
        2.        Enrichment: add some information based on prior events
             –         E.g. customer purchase: add the customer history to the
                       purchase event
        3.        Aggregation: add some information based on prior events
             –         E.g. customer purchase history: track the sum total of order
                       values for a customer
        4.        Routing: based on the purchase type pass the event on to
                  the appropriate service
             –         E.g. customer purchase event: pass on to a provisioning
71
                       service based on the type of product / product classification
epts
     event processing technical society




Aggregation Pattern: Classification (1)

        • Alternative Names:
             – Event Summarization, Composite event, Complex event
        • EPTS Reference Architecture:
             – "During complex event detection, combining events to
                  provide new or useful information, such as trend
                  information and event statistics. Similar to event
                  consolidation "
        • EPIA:
             – Aggregate (EPA): a transformation EPA that takes as
               input a collection of events and creates a single derived
               event by applying a function over the input events.
72
epts
     event processing technical society




Aggregation Pattern: Description (2)
        • Role / Explanation
             – Aggregation is used to describe several events as a single
               composite event, generally summarizing a metric of a set of
               component events.
             – For example: generate a (composite) event that contains the
               medium price of a stock over a 10 minute stream of events.
        • Associated Business Rule Specification
             – a summarisation (to be enforced during event processing):
               The <aggregation fn> of <event entities> that
               are <selection constraint>
               must have
                                                  Aggregation
               <some constraint>.      event                       event
                                          event     f(events)
73
                                           event
epts
     event processing technical society




Aggregation Pattern: Structure (3)

        • EPTS Glossary comparison
             – The term aggregate event is sometimes used for some
               forms of composite or derived event.
             – Composite event: a derived, complex event
                  • is created by combining base events using a specific set of
                    event constructors such as disjunction, conjunction,
                    sequence, etc.
                  • always includes the base (member) events from which it is
                    derived.
             – Derived event (/synthesized event): an event that is
               generated as a result of applying a method or process to
               one or more other events.
74
epts
     event processing technical society




Aggregation Pattern: Implementations (4)

        • TIBCO, Oracle, Prova and IBM implementations for
             this pattern




                                                    Aggregation
                                          event                   event
                                           event     f(events)
                                            event




75
epts
     event processing technical society



Aggregation Pattern: Implementation 1
TIBCO BusinessEvents: Rules
        • General pattern:                        Aggregation
             <declare> event,                       object
                                                                event
             aggregation-object
                                                           Identify
             <if>                                         event for
             event-member-of-aggregation                 aggregation
             <then>
             update-aggregation-object                   Aggregate
             with event-data                             Operation
                                                          on event

        • Effect: construct aggregation                    Updated
             (event) object for each incoming event       Aggregation
76                                                          Object
epts
     event processing technical society



Aggregation Pattern: Implementation 2a
Oracle CEP: queries
        • General pattern template:            Aggregation
                                                 object
                                                             event
             SELECT <summary-property>
             FROM <STREAM>[WIN]
                                                        Identify
             WHERE <predicate>
                                                       event for
             GROUP BY <aggregation-identity>          aggregation

        • Effect: summarizes the properties of        Aggregate
             several simple event into a new          Operation
             complex event                             on event

                                                        Updated
                                                       Aggregation
77                                                       Object
epts
     event processing technical society



Aggregation Pattern: Implementation 2b
Oracle CEP: model

        • Scenario:
             – Output the average bid and ask price of a stock in the
               last 10 seconds.
        • Application Model




78
epts
     event processing technical society



Aggregation Pattern: Implementation 2c
Oracle CEP: queries



                 SELECT symbol, AVG(bid), AVG(ask)


                 FROM
                    StockTickStream [RANGE 10 SECONDS]


                 GROUP BY
                    symbol



79
epts
     event processing technical society



Aggregation Pattern: Implementation 2d
Oracle CEP: table metaphor


                Time                 Input                      Output

                1s                        {“AAA”, 10.0, 12.0}     {“AAA”, 10.0, 12.0}


                4s                        {“AAA”, 12.0, 14.0}     {“AAA”, 11.0, 13.0}

                                                                  {“AAA”, 11.0, 13},
                9s                         {“BBB”, 4.0, 5.0}
                                                                   {“BBB”, 4.0, 5.0}

                15s                       {“BBB”, 8.0, 10.0}       {“BBB”, 6.0, 7.5}
                                                                 > 10s
                20s                                               {“BBB”, 8.0, 10.0}


80
epts
       event processing technical society



 Aggregation Pattern: Implementation 3a
 Prova: rules                         Aggregation
                                                    object
                                                                 event
• General pattern template:
 <rule_head>     :-                               Check & Update
      <Create Aggregator>,                       Timer / Size Count
      @group(<reaction group>)
      @timer|size(Start,Interval,Aggregator)        Identify event
      rcvMsg <EventPattern> [<Aggregate Operation>]. aggregation
                                                   for
                                                                      or
 <rule_head> :-                                    Aggregate
  @or(<reaction group>)                            Operation
  rcvMsg <Aggregator>,                              on event
  ... <consume Aggregator>.

• Effect: Repeated incremental                 Updated
                                              Aggregation
                                                              Consume
                                                             Aggregation
 aggregations over new events                   Object         Object

 81
epts
 event processing technical society


Aggregation Pattern: Implementation 3b
Prova: Example with Time Counter
 % This reaction operates indefinitely. When the timer
 elapses (after 25 ms), the groupby map Counter is sent
 as part of the aggregation event and consumed in or
 group, and the timer is reset back to the second
 argument of @timer.

groupby_rate() :-
  Counter = ws.prova.eventing.MapCounter(), % Aggr. Obj.
  @group(g1) @timer(25,25,Counter) % timer every 25 ms
   rcvMsg(XID,stream,From,inform,tick(S,P,T)) % event
       [IM=T,Counter.incrementAt(IM)]. % aggr. operation

groupby_rate() :-
  % receive the aggregation counter in the or reaction
  @or(g1) rcvMsg(XID,self,From,or,[Counter]),
   ... <consume the Counter aggreation object>.
epts
  event processing technical society

Aggregation Pattern: Implementation 4a
IBM WebSphere Decision Server: Business Events
component
• General Pattern:
                                                 event   event
  Define how the
  events should be
  accumulated
                                                 Add to array
 ... then when
 processing the
 events, use the                                  Intermediate
 relevant                                         Object Array
 aggregator

• Effect of the example: sum up the value of a    Operation
  customer‟s purchases over a week               on aggregate
  83
epts
   event processing technical society




Aggregation Pattern: Implementation 4b
IBM InfoSphere Streams: Streams Processing Language
 • Example:
       // sliding window
       stream<Beat, tuple<int32 maxSalary, uint32 ageOfMaxSalary> >
               Agg4 = Aggregate(Beat)
       {
           window
               Beat : sliding, time(10.5), count(10);
           output
               Agg4 : maxSalary = Max(salary), ageOfMaxSalary = ArgMax(salary, age);
       }




 • Effect of example: Aggregate the salary values in the
   stream within a time-based window.
  84
epts
     event processing technical society




Some EP Patterns to review in more detail
        1.        Filter: filter out all events that have some property in their
                  payload / data
             –         E.g. customer purchases: filter out those with values < $100
        2.        Enrichment: add some information based on prior events
             –         E.g. customer purchase: add the customer history to the
                       purchase event
        3.        Aggregation: add some information based on prior events
             –         E.g. customer purchase history: track the sum total of order
                       values for a customer
        4.        Routing: based on the purchase type pass the event on to
                  the appropriate service
             –         E.g. customer purchase event: pass on to a provisioning
85
                       service based on the type of product / product classification
epts
     event processing technical society




Routing Pattern: Classification (1)

        • Alternative Names:
             – Event Summarization, Composite event, Complex event
        • EPTS Reference Architecture:
             – "During event reaction, event routing is the process by
                  which an event is redirected to some process,
                  computation element, or other event sink. "
        • EPIA: no direct analogy, possibly maps to split /
             compose / project
             – Project (EPA): a translate EPA that takes an input event
               and creates a single derived event containing a subset
               of the attributes of the input event.
86
epts
     event processing technical society




Routing Pattern: Description (2)
        • Role / Explanation
             – Routing is used to describe the process of adding one or
               more new destinations to an event.
             – For example: route an input event to the appropriate
               specialist agent for that event type.
        • Associated Business Rule Specification
             – a constraint (to be enforced during event processing):
               The <event> that
                                                   Other data
               satisfies <selection constraint>     or event
               must be assigned to
               <destination>.
                                                        Routing
                                          event                          event
                                                  dest = f(event+data)
87
epts
     event processing technical society




Routing Pattern: Structure (3)

        • EPTS Glossary comparison
             – Routing (a process on events) is not defined, but is
               related to associating an event to an event sink.
             – Event sink (event consumer) is an entity that receives
               events.
                  •    Examples:
                  •    ƒ
                       Software module
                  •    ƒ
                       Database
                  •    ƒ
                       Dashboard
                  •    ƒ
                       Person


88
epts
     event processing technical society




Routing Pattern: Implementations (4)

        • TIBCO, Oracle, Prova and IBM implementations for
             this pattern


                                                 Other data
                                                  or event


                                                 Routing
                                   event                          event
                                           dest = f(event+data)




89
epts
     event processing technical society



Routing Pattern: Implementation 1
TIBCO BusinessEvents: Rules
        • General pattern:                       Routing
             <declare> event,                     data
                                                               event
             routing-data
                                                       Identify
             <if>                                     event for
             event-matches-routing-data                 routing
             <then>
                                                   Create routed
             create-new-routed-event and
                                                      event
             send-new-routed-event
                                                           event
        • Effect: match events to routing rule
             and construct / send new event
90
epts
     event processing technical society



Routing Pattern: Implementation 2a
Oracle CEP: query pattern
        • General pattern template:               Routing
                                                   data
                                                               event
             Route 1: <filtering pattern> query
             Route 2: <filtering pattern> query
                                                         Identify
             Route n: <filtering pattern> query
                                                        event for
        • Effect: Different predicates                    routing
             associated to different queries
             evaluate and select appropriate
                                                   Select destination
             destinations


                                                            event

91
epts
     event processing technical society



Routing Pattern: Implementation 2b
Oracle CEP: model

        • Scenario:
             – Output stocks whose symbols start with „a-m‟ to
               destination 1, and whose symbols start with „n-z‟ to
               destination 2.
        • Application Model




92
epts
     event processing technical society



Routing Pattern: Implementation 2c
Oracle CEP: queries

             Destination-1:
                  SELECT *
                  FROM StockTickStream
                  WHERE symbol.matches(“^[a-m]”)


             Destination-2:
                  SELECT *
                  FROM StockTickStream
                  WHERE symbol.matches(“^[n-z]”)

93
epts
     event processing technical society



Routing Pattern: Implementation 3                        event
Prova: rules
        • General pattern template:                     Identify
                                                       event for
             rcvMsg <Event Msg Pattern> :-               routing
                  <routing decisions>,
                  sendMsg <route event >.        Routing         Routing
                                                 Decisions        data
             <routing decision rule 1>   :-
                  <decision conditions>.
                                                      Create routed
              <routing decision rule 2> :-               event
                   <decision conditions>.
                  ...
                                                             Route
                                              event
                                                             event
        • Effect: Events are routed
             according to the routing decision
94
             rules
epts
  event processing technical society


Routing Pattern: Implementation 3b
Prova: Example with Agent (Sub-) Conversations
rcvMsg(XID,esb,From,query-ref,buy(Product) :-
 routeTo(Agent,Product), % derive processing agent
 % send order to Agent in new subconversation SID2
 sendMsg(SID2,esb,Agent,query-ref,order(From,
  Product)),
 % receive confirmation from Agent for Product order
 rcvMsg(SID2,esb,Agent,inform-ref,oder(From, Product)).

% route to event processing agent 1 if Product is
  luxury
routeTo(epa1,Product) :- luxury(Product).
% route to epa 2 if Product is regular
routeTo(epa2,Product) :- regular(Product).

% a Product is luxury if the Product has a value over …
luxury(Product) :- price(Product,Value), Value >=
  10000.
% a Product is regular if the Product ha a value below
epts
    event processing technical society


Routing Pattern: Implementation 4
IBM: Business Events component of WebSphere
Decision Server
                                                   Routing
                                                                  event
• Example: Define an interaction set,            information
    which will send the request to supplier A,
    when the purchase indicates supply
    from Supplier A                                     Identify
•                                                         event
                                                         routing


                                                          event

• Effect of example:
    Events are routed as required
                                                       Consume
    96                                                   event
epts
     event processing technical society




Agenda

        • Introduction to the EPTS Functional Reference
             Architecture, and what it contains
        • Mapping Functions to Patterns: some sample
             Event Processing Patterns
        • EPTS member pattern implementation examples
        • Summary and future work on EP Patterns




97
epts
event processing technical society




   Another Pattern Type

                  Other patterns may be required for
                  implementation, such as handling
                  produced-consumer interactions
epts
     event processing technical society




Application-Time Pattern

        • Problem:
             – Need to use application‟s view of time, instead of CPU
               wall-clock.
             – This is particularly useful when events originate from
               different machines and need some way of synchronizing.
        • Scenario:
             – Again, consider bid and ask stream…
             – However, bid and ask requests are time-stamped on
               trader at the time that the order is placed.




99
epts
      event processing technical society




Application-Time Pattern

         • Scenario:
              – Seller places two ask requests respectively at time
                8:00:00 and 8:00:12
              – Buyer places one bid request at time 8:00:11




100
epts
      event processing technical society




Application-Time Pattern

         • Scenario:
              – Remember that we want to correlate using a 10 seconds
                window, as anything older is stale…
              – Hence first event from seller should not be considered,
                and we should correlate the ask price of 11.0 with the bid
                price of 9.5




101
epts
      event processing technical society




Application-Time Pattern

         • Scenario:
              – However, the reality is that the first two events could
                arrive together in a burst in the exchange, and the third
                could be delayed in the cloud…




102
epts
      event processing technical society




Application-Time Pattern

         • Scenario:
              – In this case, bid price of 9.5 would correlate to ask price of 10.0
                and the exchange would loose money as the spread is lower…




103
epts
      event processing technical society



Application Time-stamped Pattern: Implementation 1
Oracle CEP

       • Solution:
            – The query does not change…
            – However STREAMS must be configured to use
              application time-stamps based upon some event
              property, instead of having events being system-
              timestamped…




104
epts
  event processing technical society



Application Time-stamped Pattern: Implementation 2
IBM Business Monitor
• Example: Define a correlation expression



                                                  „Order Placed‟   „Order Complete‟

                                                      event             event
• Effect of example:
 The duration of a customer order can be obtained from
 the timestamps on start and end events
                                                              Correlate


                                                              Calculate
                                                              Duration
 105
epts
      event processing technical society



Application Time Pattern: Implementation 3
Prova: rules
• General pattern template:
                                                 Event

rcvMsg <Event Msg Pattern(s)> :-
  <add application timestamp to event>,
  sendMsg <Adjusted Event>.
                                             adjust event

• Effect: Any timestamp can be adjusted
  (for time computations)
                                                Updated
                                                 Event




106
epts
  event processing technical society



Application Time-stamped Pattern: Implementation 4
TIBCO BusinessEvents: PreProcessor Pattern
• General Pattern: adjust Event metadata (e.g.
 Timestamp) in preprocessor for that event
                                                      Event
 – events are generally non-monotonic!
 – PreProcessor applies regardless of engine
   used!                                          Preprocess:
• Effect:                                         adjust event
 Any custom timestamp can be adjusted (for time
 computations)

  Note: If the problem is “out of order” events      Updated
                                                      Event
     (rather than synchronizing clocks) a
        different pattern may be better!

 107
epts
event processing technical society




   Use Cases

                         Illustrating using some of the patterns
epts
  event processing technical society



IBM Use Case: Utilizing Smart Meters
Large energy company improved customer responsiveness
using event processing


                                                            Use Case
                                             • Utilized events (and lack of events) from
                                              smart meters to detect service outages and
                                                              disruptions
                                              • Empowered operations team to define
                                                outage detection and response patterns
                                                      without coding or scripting


                                                Smarter Business Outcomes
                                           • Automated detection and response to service
 Challenge: Reliance on customer calls                        outages
  to detect and respond to outages and
 reduce time to recovery while providing   • Improved customer satisfaction by delivery of
    more information to the customer          estimated time to recovery alerts based on
                                                        customer preferences
epts
  event processing technical society



Energy Company Leveraging Smart Meters to Improve
Customer Responsiveness
                                                                                                      Customer
                                                                                   Business             Data
                                                                                    Events
                                                                2

                                       1
                                                                                             3
                                            Data Collection
                                            Engine (DCE)

                                                                                      WebSphere
                                                                                    Business Events
                                       1
                                                                                             4

                                            Tivoli OMNIbus,
                                           Network Manager,
                                               and Impact
                 Power                                                      Outage
                Disruption                                                Management                Customer Care
                                                                         System (OMS)             Manager(s) with IBM
                                                                                                   Business Monitor
                                                              Location, Estimated Duration
                                                                      of Disruption
                                                                                                 Alert and Assist Impacted
                                                                                                         Customers

                                                                                                     110
epts
      event processing technical society




Oracle Use-case: Capital Markets

         • Low-Latency Trading
              – Algorithm Trading
         • Trade Matching
              – High throughput
              – In-flight fraud
         • Position Capture and Aggregation
         • Risk and Analytics
              – Pre/post trade analytics
              – On-demand risk management
111
epts
      event processing technical society




TIBCO Use Case: Application Service Gateway
Telco high performance service policies using
event processing




                                                                       Excellent service quality
Mobile Services
                                                                          needed to avoid
difficult to monitor
                                                                           customer churn
     end to end

                        Auto detect and
                         fixing of issues    Service quality affects
                             for better     take-up of new services
112
                           user service
epts
 event processing technical society

Service Gateway architecture exploits event processing

                                      •   Standardized channel for
                                          monitoring subscribers
                                      •   Standardized access to services
                                          (e.g. Facebook, Google, etc)
                                      •   Consolidation of existing multiple
                                          gateways
                                      •   Allows analysis of service usage,
                                          subscriber behavior, service
                                          performance, marketing campaign
                                          success, etc
                                      •   Creates events for revenue
                                          generation and reconciliation
                                      •   Handles Common Telco
                                          Notifications - change-of-device,
                                          contract, service- allowing
                                          UpSell/Cross sell etc
epts
 event processing technical society

Service Gateway architecture exploits event processing

                                            •   Standardized channel for
                                                monitoring subscribers
                                          • Standardized access to services
                                             (e.g. Facebook, Google, etc)
                                      Security Pattern
                                          • Consolidation of existing multiple
                                             gateways
                                      Decision / Policy
                                          • Allows
                                          Pattern analysis of service usage,
                                             subscriber behavior, service
                                            performance, marketing campaign
                                            success, etc
                                      Routing Pattern
                                          • Creates events for revenue
                                            generation and reconciliation
                                      Transformation
                                          • Handles Common Telco
                                         Patterns
                                            Notifications - change-of-device,
                                                contract, service- allowing
                                                UpSell/Cross sell etc
epts
      event processing technical society




Agenda

         • Introduction to the EPTS Functional Reference
              Architecture, and what it contains
         • Mapping Functions to Patterns: some sample
              Event Processing Patterns
         • EPTS member pattern implementation examples
         • Summary and future work on EP Patterns




115
epts
  event processing technical society



Summary: Realization Patterns / Idioms

                                Event Reaction
                         Assessment, Routing, Prediction,
                               Discovery, Learning




                                                                       State Management
                                                                0..*
                                 Complex Event Detection
                                 Consolidation, Composition,
                                        Aggregation
                                                                0..*
                                    Event Analysis
                            Analytics, Transforms, Tracking,
                             Scoring, Rating, Classification
                                                                0..*
                                Event Preparation
                        Identification, Selection, Filtering,
                             Monitoring, Enrichment
                                                                0..*

             Event Production                       Event Consumption
              Application Time                       Dashboard, Apps,
            Publication, Retrieval                    External Reaction
epts
 event processing technical society



Categorization of Event Processing Patterns




                                      Source:Paschke, A.: Design Patterns for Complex Event Processing, DEBS'08, Rome, Italy, 2008
                                                                                                 http://arxiv.org/abs/0806.1100v1
epts
    event processing technical society   Categorization of Patterns
•      Categorization according to Good and Bad Solutions
     – CEP Patterns
     – CEP Anti-Patterns
•      Categorization according to the Abstraction Level
     –    Guidelines and Best Practices
     –    Management patterns
     –    Architecture patterns
     –    Design patterns
     –    Mapping patterns
     –    Idioms / Realization patterns
     –    Smells / Refactoring patterns
•      Categorization according to the Intended Goal
     –    Adoption patterns
     –    Business patterns
     –    Integration patterns
     –    Composite patterns:
     –    …
•    Categorization according to the Management Level
     – Strategic patterns
     – Tactical patterns
                                                    Source:Paschke, A.: A Semantic Design Pattern Language for Complex Event Processing,
     – Operational patterns                    Intelligent Event Processing, Papers from the 2009 AAAI Spring Symposium (2009) , p. 54-60.
                                                                    http://www.aaai.org/Papers/Symposia/Spring/2009/SS-09-05/SS09-05-010.pdf
epts
    event processing technical society




Next Steps
•   Continue evolution of EP Reference Architecture
    – Eg Logical Architecture vs Systems Description
•   Evolve the EP Pattern <<metapattern>> to be more useful for
    developers and tools
•   Expand coverage to all Functions as Patterns, + extend Functions as
    necessary
•   Planned Outputs:
    – EPTS-RA Description white paper
    – Wiley Book "Pattern Oriented Software Architecture: Architectures,
      Models and Patterns for Event Processing" (already in preparation)
    – Input for Event Processing Metamodels and associated standards
epts
      event processing technical society




Summary

         • Reference Architecture provides a common set of
              EP Functions that may be mapped
              to other EP ontologies / taxonomies
         • Provides the basis for discussing EP Patterns
         • EP Patterns can be semantic, design or runtime
              focused
         • Further evolution of EP Pattern and examples is
              likely!



120
epts
      event processing technical society




            Thank you !



                                           Acknowledgment to the Event Processing Technical Society
                                           Reference Architecture working group members




121
epts
    event processing technical society




References
•   EPTS Glossary
     http://www.ep-ts.com/component/option,com_docman/task,doc_download/gid,66/Itemid,84/

More Related Content

Viewers also liked

Pattern Driven Enterprise Architecture
Pattern Driven Enterprise ArchitecturePattern Driven Enterprise Architecture
Pattern Driven Enterprise ArchitectureWSO2
 
DSI Corporate Overview
DSI Corporate OverviewDSI Corporate Overview
DSI Corporate OverviewJoel Limerick
 
Event Driven Architecture (EDA) Reference Architecture | Anbu Krishnaswamy
Event Driven Architecture (EDA) Reference Architecture | Anbu KrishnaswamyEvent Driven Architecture (EDA) Reference Architecture | Anbu Krishnaswamy
Event Driven Architecture (EDA) Reference Architecture | Anbu KrishnaswamyBob Rhubart
 
RuleML2011 CEP Standards Reference Model
RuleML2011 CEP Standards Reference ModelRuleML2011 CEP Standards Reference Model
RuleML2011 CEP Standards Reference ModelPaul Vincent
 
Pattern driven Enterprise Architecture
Pattern driven Enterprise ArchitecturePattern driven Enterprise Architecture
Pattern driven Enterprise ArchitectureWSO2
 
Architecture Patterns - Open Discussion
Architecture Patterns - Open DiscussionArchitecture Patterns - Open Discussion
Architecture Patterns - Open DiscussionNguyen Tung
 
Software engineering : Layered Architecture
Software engineering : Layered ArchitectureSoftware engineering : Layered Architecture
Software engineering : Layered ArchitectureMuhammed Afsal Villan
 
Api gateway : To be or not to be
Api gateway : To be or not to beApi gateway : To be or not to be
Api gateway : To be or not to beJaewoo Ahn
 
Facebook Architecture - Breaking it Open
Facebook Architecture - Breaking it OpenFacebook Architecture - Breaking it Open
Facebook Architecture - Breaking it OpenHARMAN Services
 
Architectural styles and patterns
Architectural styles and patternsArchitectural styles and patterns
Architectural styles and patternsHimanshu
 
facebook architecture for 600M users
facebook architecture for 600M usersfacebook architecture for 600M users
facebook architecture for 600M usersJongyoon Choi
 
Fundamentals Of Software Architecture
Fundamentals Of Software ArchitectureFundamentals Of Software Architecture
Fundamentals Of Software ArchitectureMarkus Voelter
 
Architectural styles and patterns
Architectural styles and patternsArchitectural styles and patterns
Architectural styles and patternsdeep sharma
 
Semantic Complex Event Processing
Semantic Complex Event ProcessingSemantic Complex Event Processing
Semantic Complex Event ProcessingAdrian Paschke
 

Viewers also liked (15)

Pattern Driven Enterprise Architecture
Pattern Driven Enterprise ArchitecturePattern Driven Enterprise Architecture
Pattern Driven Enterprise Architecture
 
DSI Corporate Overview
DSI Corporate OverviewDSI Corporate Overview
DSI Corporate Overview
 
Event Driven Architecture (EDA) Reference Architecture | Anbu Krishnaswamy
Event Driven Architecture (EDA) Reference Architecture | Anbu KrishnaswamyEvent Driven Architecture (EDA) Reference Architecture | Anbu Krishnaswamy
Event Driven Architecture (EDA) Reference Architecture | Anbu Krishnaswamy
 
RuleML2011 CEP Standards Reference Model
RuleML2011 CEP Standards Reference ModelRuleML2011 CEP Standards Reference Model
RuleML2011 CEP Standards Reference Model
 
Pattern driven Enterprise Architecture
Pattern driven Enterprise ArchitecturePattern driven Enterprise Architecture
Pattern driven Enterprise Architecture
 
Architecture Patterns - Open Discussion
Architecture Patterns - Open DiscussionArchitecture Patterns - Open Discussion
Architecture Patterns - Open Discussion
 
Software engineering : Layered Architecture
Software engineering : Layered ArchitectureSoftware engineering : Layered Architecture
Software engineering : Layered Architecture
 
Api gateway : To be or not to be
Api gateway : To be or not to beApi gateway : To be or not to be
Api gateway : To be or not to be
 
Facebook Architecture - Breaking it Open
Facebook Architecture - Breaking it OpenFacebook Architecture - Breaking it Open
Facebook Architecture - Breaking it Open
 
Architectural styles and patterns
Architectural styles and patternsArchitectural styles and patterns
Architectural styles and patterns
 
facebook architecture for 600M users
facebook architecture for 600M usersfacebook architecture for 600M users
facebook architecture for 600M users
 
Fundamentals Of Software Architecture
Fundamentals Of Software ArchitectureFundamentals Of Software Architecture
Fundamentals Of Software Architecture
 
Architectural styles and patterns
Architectural styles and patternsArchitectural styles and patterns
Architectural styles and patterns
 
Layered Software Architecture
Layered Software ArchitectureLayered Software Architecture
Layered Software Architecture
 
Semantic Complex Event Processing
Semantic Complex Event ProcessingSemantic Complex Event Processing
Semantic Complex Event Processing
 

Similar to EPTS DEBS2011 Event Processing Reference Architecture and Patterns Tutorial v1 2

Event Driven Architecture : Managing Business Dynamics for Adaptive Enterprise
Event Driven Architecture : Managing Business Dynamics for Adaptive EnterpriseEvent Driven Architecture : Managing Business Dynamics for Adaptive Enterprise
Event Driven Architecture : Managing Business Dynamics for Adaptive EnterpriseDassana Wijesekara
 
Event Driven Architecture : Managing Business Dynamics for an Adaptive Enterp...
Event Driven Architecture : Managing Business Dynamics for an Adaptive Enterp...Event Driven Architecture : Managing Business Dynamics for an Adaptive Enterp...
Event Driven Architecture : Managing Business Dynamics for an Adaptive Enterp...Dassana Wijesekara
 
The Essentials of Great Search Design (ECIR 2010)
The Essentials of Great Search Design (ECIR 2010)The Essentials of Great Search Design (ECIR 2010)
The Essentials of Great Search Design (ECIR 2010)Vegard Sandvold
 
Detailed design
Detailed designDetailed design
Detailed designjsokohl
 
Detailed design: Nailing it Down
Detailed design: Nailing it DownDetailed design: Nailing it Down
Detailed design: Nailing it Downjsokohl
 
DashMash: a Mashup Environment for End User Development
DashMash: a Mashup Environment for End User DevelopmentDashMash: a Mashup Environment for End User Development
DashMash: a Mashup Environment for End User DevelopmentMatteo Picozzi
 
Saf08 Growing Architects Kevin Francis
Saf08 Growing Architects   Kevin FrancisSaf08 Growing Architects   Kevin Francis
Saf08 Growing Architects Kevin FrancisKevin Francis
 
Leadership Symposium on Digital Media in Healthcare
Leadership Symposium on Digital Media in HealthcareLeadership Symposium on Digital Media in Healthcare
Leadership Symposium on Digital Media in Healthcaresetstanford
 
Icalt2010 hoel hollins
Icalt2010 hoel hollinsIcalt2010 hoel hollins
Icalt2010 hoel hollinsTore Hoel
 
Architecture solution architecture method
Architecture solution architecture methodArchitecture solution architecture method
Architecture solution architecture methodChris Eaton
 
Correlation Architecture
Correlation ArchitectureCorrelation Architecture
Correlation Architecturesboray
 
Welcome To Etech
Welcome To EtechWelcome To Etech
Welcome To EtechTom Brandt
 
Systems engineering for project managers - what you need to know
Systems engineering for project managers - what you need to knowSystems engineering for project managers - what you need to know
Systems engineering for project managers - what you need to knowAssociation for Project Management
 
Application Lifecycle Management & VSTS
Application Lifecycle Management & VSTSApplication Lifecycle Management & VSTS
Application Lifecycle Management & VSTSMicrosoft Iceland
 
From Process Design to Process Automation
From Process Design to Process AutomationFrom Process Design to Process Automation
From Process Design to Process AutomationJohan den Haan
 
Iasa UK Archimate Overview
Iasa UK Archimate OverviewIasa UK Archimate Overview
Iasa UK Archimate OverviewIasa UK
 
IDF 2011: ODCA & Developing a Usage Model Roadmap for Cloud Computing
IDF 2011: ODCA & Developing a Usage Model Roadmap for Cloud ComputingIDF 2011: ODCA & Developing a Usage Model Roadmap for Cloud Computing
IDF 2011: ODCA & Developing a Usage Model Roadmap for Cloud ComputingOpen Data Center Alliance
 
Open Data Center Alliance Workgroups, Usage Models and Roadmap Structure
Open Data Center Alliance Workgroups, Usage Models and Roadmap StructureOpen Data Center Alliance Workgroups, Usage Models and Roadmap Structure
Open Data Center Alliance Workgroups, Usage Models and Roadmap StructureOpen Data Center Alliance
 
01 necto introduction_ready
01 necto introduction_ready01 necto introduction_ready
01 necto introduction_readywww.panorama.com
 

Similar to EPTS DEBS2011 Event Processing Reference Architecture and Patterns Tutorial v1 2 (20)

Brochure for pmvt
Brochure for pmvtBrochure for pmvt
Brochure for pmvt
 
Event Driven Architecture : Managing Business Dynamics for Adaptive Enterprise
Event Driven Architecture : Managing Business Dynamics for Adaptive EnterpriseEvent Driven Architecture : Managing Business Dynamics for Adaptive Enterprise
Event Driven Architecture : Managing Business Dynamics for Adaptive Enterprise
 
Event Driven Architecture : Managing Business Dynamics for an Adaptive Enterp...
Event Driven Architecture : Managing Business Dynamics for an Adaptive Enterp...Event Driven Architecture : Managing Business Dynamics for an Adaptive Enterp...
Event Driven Architecture : Managing Business Dynamics for an Adaptive Enterp...
 
The Essentials of Great Search Design (ECIR 2010)
The Essentials of Great Search Design (ECIR 2010)The Essentials of Great Search Design (ECIR 2010)
The Essentials of Great Search Design (ECIR 2010)
 
Detailed design
Detailed designDetailed design
Detailed design
 
Detailed design: Nailing it Down
Detailed design: Nailing it DownDetailed design: Nailing it Down
Detailed design: Nailing it Down
 
DashMash: a Mashup Environment for End User Development
DashMash: a Mashup Environment for End User DevelopmentDashMash: a Mashup Environment for End User Development
DashMash: a Mashup Environment for End User Development
 
Saf08 Growing Architects Kevin Francis
Saf08 Growing Architects   Kevin FrancisSaf08 Growing Architects   Kevin Francis
Saf08 Growing Architects Kevin Francis
 
Leadership Symposium on Digital Media in Healthcare
Leadership Symposium on Digital Media in HealthcareLeadership Symposium on Digital Media in Healthcare
Leadership Symposium on Digital Media in Healthcare
 
Icalt2010 hoel hollins
Icalt2010 hoel hollinsIcalt2010 hoel hollins
Icalt2010 hoel hollins
 
Architecture solution architecture method
Architecture solution architecture methodArchitecture solution architecture method
Architecture solution architecture method
 
Correlation Architecture
Correlation ArchitectureCorrelation Architecture
Correlation Architecture
 
Welcome To Etech
Welcome To EtechWelcome To Etech
Welcome To Etech
 
Systems engineering for project managers - what you need to know
Systems engineering for project managers - what you need to knowSystems engineering for project managers - what you need to know
Systems engineering for project managers - what you need to know
 
Application Lifecycle Management & VSTS
Application Lifecycle Management & VSTSApplication Lifecycle Management & VSTS
Application Lifecycle Management & VSTS
 
From Process Design to Process Automation
From Process Design to Process AutomationFrom Process Design to Process Automation
From Process Design to Process Automation
 
Iasa UK Archimate Overview
Iasa UK Archimate OverviewIasa UK Archimate Overview
Iasa UK Archimate Overview
 
IDF 2011: ODCA & Developing a Usage Model Roadmap for Cloud Computing
IDF 2011: ODCA & Developing a Usage Model Roadmap for Cloud ComputingIDF 2011: ODCA & Developing a Usage Model Roadmap for Cloud Computing
IDF 2011: ODCA & Developing a Usage Model Roadmap for Cloud Computing
 
Open Data Center Alliance Workgroups, Usage Models and Roadmap Structure
Open Data Center Alliance Workgroups, Usage Models and Roadmap StructureOpen Data Center Alliance Workgroups, Usage Models and Roadmap Structure
Open Data Center Alliance Workgroups, Usage Models and Roadmap Structure
 
01 necto introduction_ready
01 necto introduction_ready01 necto introduction_ready
01 necto introduction_ready
 

Recently uploaded

AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101vincent683379
 
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptxIOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptxAbida Shariff
 
The architecture of Generative AI for enterprises.pdf
The architecture of Generative AI for enterprises.pdfThe architecture of Generative AI for enterprises.pdf
The architecture of Generative AI for enterprises.pdfalexjohnson7307
 
Knowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and backKnowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and backElena Simperl
 
Strategic AI Integration in Engineering Teams
Strategic AI Integration in Engineering TeamsStrategic AI Integration in Engineering Teams
Strategic AI Integration in Engineering TeamsUXDXConf
 
Transforming The New York Times: Empowering Evolution through UX
Transforming The New York Times: Empowering Evolution through UXTransforming The New York Times: Empowering Evolution through UX
Transforming The New York Times: Empowering Evolution through UXUXDXConf
 
Motion for AI: Creating Empathy in Technology
Motion for AI: Creating Empathy in TechnologyMotion for AI: Creating Empathy in Technology
Motion for AI: Creating Empathy in TechnologyUXDXConf
 
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...CzechDreamin
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaRTTS
 
Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessUXDXConf
 
Introduction to Open Source RAG and RAG Evaluation
Introduction to Open Source RAG and RAG EvaluationIntroduction to Open Source RAG and RAG Evaluation
Introduction to Open Source RAG and RAG EvaluationZilliz
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераMark Opanasiuk
 
Designing for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastDesigning for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastUXDXConf
 
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...CzechDreamin
 
AI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří KarpíšekAI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří KarpíšekCzechDreamin
 
"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor TurskyiFwdays
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
 
Optimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityOptimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityScyllaDB
 
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaCzechDreamin
 

Recently uploaded (20)

AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101
 
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptxIOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
 
The architecture of Generative AI for enterprises.pdf
The architecture of Generative AI for enterprises.pdfThe architecture of Generative AI for enterprises.pdf
The architecture of Generative AI for enterprises.pdf
 
Knowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and backKnowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and back
 
Strategic AI Integration in Engineering Teams
Strategic AI Integration in Engineering TeamsStrategic AI Integration in Engineering Teams
Strategic AI Integration in Engineering Teams
 
Transforming The New York Times: Empowering Evolution through UX
Transforming The New York Times: Empowering Evolution through UXTransforming The New York Times: Empowering Evolution through UX
Transforming The New York Times: Empowering Evolution through UX
 
Motion for AI: Creating Empathy in Technology
Motion for AI: Creating Empathy in TechnologyMotion for AI: Creating Empathy in Technology
Motion for AI: Creating Empathy in Technology
 
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
 
Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
 
Introduction to Open Source RAG and RAG Evaluation
Introduction to Open Source RAG and RAG EvaluationIntroduction to Open Source RAG and RAG Evaluation
Introduction to Open Source RAG and RAG Evaluation
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 
Designing for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastDesigning for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at Comcast
 
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
 
AI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří KarpíšekAI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří Karpíšek
 
"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
 
Optimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityOptimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through Observability
 
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara Laskowska
 

EPTS DEBS2011 Event Processing Reference Architecture and Patterns Tutorial v1 2

  • 1. epts event processing technical society Tutorial v1.2 epts event processing technical society Content by: members of the EPTS Reference Architecture Working Group Event Processing Adrian Paschke (Freie Universitaet Berlin) Paul Vincent Reference Architecture (TIBCO Software) Catherine Moxey and Design Patterns (IBM) Alex Alves (Oracle)
  • 2. epts event processing technical society Tutorial • Event Processing is an increasingly “mainstream” view in IT, if not widely in production • Event Processing Functional Reference Architecture (FRA) describes the functions involved in event processing • Functions in the FRA can themselves be described as event processing patterns • Most technology providers can support the FRA and its associated design patterns 2
  • 3. epts event processing technical society Agenda • Introduction to the EPTS Functional Reference Architecture, and what it contains • Mapping Functions to Patterns: some sample Event Processing Patterns • EPTS member pattern implementation examples • Summary and future work on EP Patterns 3
  • 4. epts event processing technical society Introduction to Event Processing • Event-centric view of IT • Events are – Sent and Received – Aggregated, Transformed into Data, Deleted – Processed in queries, rules etc – Cause actions like processes, service invocations, etc 4
  • 5. epts event processing technical society Introduction to Event Processing Event Processing, Complex Event Processing, Event Stream Processing Event Processing Rules, etc. Event Type Definitions, Event Abstraction, Event Pattern Detection, Event Composition Event Management etc. etc. Events Derived Events Event Producer Event Consumer (Event Source, (Event sink, event Event Emitter) handler, event listener) Design time Run time Administration
  • 6. epts event processing technical society Motivation and Benefits • Motivation – Event Processing is evolving and exploiting many technologies – Potential adopters (stakeholders) need a reference to understand suppliers‟ architectures and solutions. • Benefits – a Reference Architecture predefines customizable abstract frames of reference for specific stakeholder concerns and application domains. • Aids reuse of successful EP architectures for frequently occurring EP design problems – Underlying Architecture is a Reference Model that defines the terminology and components in Event Processing architectures
  • 7. epts event processing technical society Reference Architecture Viewpoints Viewpoint Viewpoint Element Engineering EP Managing EP Business with EP Architecture Architecture Architecture Concepts How to How to apply? How to utilize / sell / implement? own? Stakeholders Architects / Project Manager Decision Maker, Engineers Customer, Provider Concerns Effective Operational Strategic and construction and Management tactical deployment management Techniques / Modeling, IT (service/appl) Monitoring, Languages Engineering management, Enterprise Decision project Management, management Governance 10
  • 8. epts event processing technical society Stakeholders and Viewpoints covered Viewpoints (for CEP / stateful EP) Stakeholder Component Functional Decision Maker [Not applicable] Inputs, Outputs and / End User Processing Requirements View Architect / Solution Components Functions carried out in Designer View CEP View TBD (2012?) Focus so far 11
  • 9. epts event processing technical society Functional Reference Architecture • Architect and Developer perspective – includes the 3 main functions (development, run-time and administration), – targets primarily the automated event processing operations • Run-time functions in 2 main groups: – the event infrastructure (sources and consumers) external to the event processor under consideration, – the event processor.
  • 10. epts event processing technical society Functional View source: definitions of EP Event Processing, Complex Event Processing, Event Stream Processing Event Processing Rules, etc. Event Type Definitions, Event Abstraction, Event Pattern Detection, Event Composition Event Management etc. etc. Events Derived Events Event Producer Event Consumer (Event Source, (Event sink, event Event Emitter) handler, event listener) Design time Run time Administration
  • 11. epts event processing technical society Reference Architecture: Functional View Definition, Modeling, (continuous) Improvement Event Reaction Assessment, Routing, Prediction, Discovery, Learning (Pattern, Control, Rule, Query, RegEx.etc) Event Process Monitoring, Control 0..* State Management Complex Event Detection Event and Complex Event Consolidation, Composition, Aggregation 0..* Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification 0..* Event Preparation Identification, Selection, Filtering, Monitoring, Enrichment 0..* Event Production Event Consumption Publication, Dashboard, Apps, Retrieval External Reaction Design time Run time Administration
  • 12. epts event processing technical society Reference Architecture: Functional View / Runtime Event Reaction Assessment, Routing, Prediction, Discovery, Learning 0..* State Management Complex Event Detection Consolidation, Composition, Aggregation 0..* Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification 0..* Event Preparation Identification, Selection, Filtering, Monitoring, Enrichment 0..* Event Production Event Consumption Publication, Dashboard, Apps, Retrieval External Reaction Run time
  • 13. epts event processing technical society Reference Architecture: Functional View / Runtime Event Production: the source of events for event processing. • Event Publication: As a part of event production, events may be published onto a communication mechanism (eg event bus) for use by event consumers (including participants in event processing). This is analogous to a "push" system for obtaining events. • Event Retrieval: As a part of event production, events may be explicitly retrieved from some detection system. This is analogous to a "pull" system for obtaining events. Event Production Event Consumption Publication, Dashboard, Apps, Retrieval External Reaction
  • 14. epts event processing technical society Reference Architecture: Functional View / Runtime Event Consumption: the process of using events from event publication and processing. Event processing itself can be an event consumer, although for the purposes of the reference architecture, event consumers are meant to indicate downstream consumers of events generated in event processing. • Dashboard: a type of event consumer that displays events as they occur to some user community. • Applications: a type of event consumer if it consumes events for its own processes. • External Reaction: caused through some event consumption, as the result of some hardware or software process. Event Production Event Consumption Publication, Dashboard, Apps, Retrieval External Reaction
  • 15. epts event processing technical society Reference Architecture: Functional View / Runtime Event Preparation: the process of preparing the event and associated payload and metadata for further stages of event processing. • Entity Identification: incoming events will need to be identified relative to prior events, such as associating events with particular sources or sensors. • Event Selection: particular events may be selected for further analysis. Different parts of event processing may require different selections of events. See also event filtering. • Event Filtering: a stream or list of events may be filtered on some payload or metadata information such that some subset is selected for further processing. • Event Monitoring: particular types of events may be monitored for selection for further processing. This may utilise specific mechanisms external to the event processing such as exploiting event production features. • Event Enrichment: events may be "enriched" through knowledge gained through previous events or data. Event Preparation Identification, Selection, Filtering, Monitoring, Enrichment
  • 16. epts event processing technical society Reference Architecture: Functional View / Runtime Event Analysis: the process of analysing suitably prepared events and their payloads and metadata for useful information. • Event Analytics: the use of statistical methods to derive additional information about an event or set of events. • Event Transforms: processes carried out on event payloads or data, either related to event preparation, analysis or processing. • Event Tracking: where events related to some entity are used to identify state changes in that entity. • Event Scoring: the process by which events are ranked using a score, usually as a part of a statistical analysis of a set of events. See also Event Analytics • Event Rating: where events are compared to others to associate some importance or other, possibly relative, measurement to the event. • Event Classification: where events are associated with some classification scheme for use in downstream processing. Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
  • 17. epts event processing technical society Reference Architecture: Functional View / Runtime Complex Event Detection: the process by which event analysis results in the creation of new event information, or the update of existing complex events. • Event Consolidation: combining disparate events together into a "main" or "primary" event. See also event aggregation. • Event Composition: composing new, complex events from existing, possibly source, events. • Event Aggregation: combining events to provide new or useful information, such as trend information and event statistics. Similar to event consolidation. Complex Event Detection Consolidation, Composition, Aggregation
  • 18. epts event processing technical society Reference Architecture: Functional View / Runtime Event Reaction: the process subsequent to event analysis and complex event detection to handle the results of analysis and detection. • Event Assessment: the process by which an event is assessed for inclusion in some process, incorporation in some other event, etc. • Event Routing: the process by which an event is redirected to some process, computation element, or other event sink. • Event Prediction: where the reaction to some event processing is that some new event is predicted to occur. • Event Discovery: where the reaction to some event processing is the disclosure of a new, typically complex, event type. • Note that event prediction is predicting some future event, usually of a known type, whereas event discovery is the uncovering of a new event type. See also event-based learning. • Event-based Learning: the reaction to some event processing that uses new event information to add to some, typically statistical-based, understanding of events. • Note that event-based learning is a specialisation of general machine learning and predictive analytics. Event Reaction Assessment, Routing, Prediction, Discovery, Learning
  • 19. epts event processing technical society Reference Architecture: Functional View / Design time Definition, Modeling, (continuous) Improvement Covers the definition, modeling, improvement / (Pattern, Control, Rule, Query, RegEx.etc) maintenance of the artifacts used in event processing: Event and Complex Event • event definitions, including event metadata and payloads, • event and event object organisations and structures, • event processing transformations / queries / rules / procedures / flows / states / decisions / expressions (although these can sometimes be considered as administrative updates in some situations) Design time
  • 20. epts event processing technical society Reference Architecture: Functional View / Administration Administrative concepts of Event Process Monitoring, Control monitoring and control. This may involve • starting and stopping the application and event processing elements, including application monitors • providing and updating security levels to event inputs and outputs (also can design-time) • management of high availability and reliability resources, such as hot standby processes • resource utilisation monitoring of the event processing components • process updates, such as how-swapping of event processing definitions to newer versions. Administration
  • 21. epts event processing technical society Agenda • Introduction to the EPTS Functional Reference Architecture, and what it contains • Mapping Functions to Patterns: some sample Event Processing Patterns • EPTS member pattern implementation examples • Summary and future work on EP Patterns 24
  • 22. epts event processing technical society Event Processing Patterns • Functions from Reference Architecture are a guide to possible low-level patterns Event Reaction Assessment, Routing, Prediction, Discovery, Learning Complex Event Detection Consolidation, Composition, Aggregation Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification Event Preparation Identification, Selection, Filtering, Monitoring, Enrichment 25
  • 23. epts event processing technical society Which Classification of Patterns to Use? • Common EP Functions? As described in the Reference Architecture • Other sources? Recent literature 26
  • 24. epts event processing technical society Comparison of EPTS RA Fns and EPIA EP Agents Event Reaction Assessment, Routing, Prediction, Discovery, Learning Complex Event Detection Consolidation, Composition, Aggregation Event Analysis Analytics, Transforms, Tracking, EPA Scoring, Rating, Classification Event Preparation Filter Transformation Pattern detect Identification, Selection, Filtering, Monitoring, Enrichment Translate Aggregate Split Compose Enrich Project 27
  • 25. epts event processing technical society Comparison of EPTS RA Fns and EPIA EP Agents Event Preparation 28
  • 26. epts event processing technical society Comparison of EPTS RA Fns and EPIA EP Agents Event Analysis 29
  • 27. epts event processing technical society Comparison of EPTS RA Fns and EPIA EP Agents Complex Event Detection 30
  • 28. epts event processing technical society Comparison of EPTS RA Fns and EPIA EP Agents Event Reaction 31
  • 29. epts event processing technical society What do we want to cover in an EP Pattern? Type / Operands Operation / synonym Operators Event Filtering: a stream or list of events may be filtered on some payload or metadata information such that some subset is selected for further processing. Rationale / Mapping to business rule EPL template 32
  • 30. epts event processing technical society EP MetaPattern version 1 • Name & Alternative Names • Classification versus other references – EPTS RA and other references e.g. “EP in Action” book • Description – Role, and Business Rule type specification / requirement • Structure – EPTS Glossary terms • Implementations 33
  • 31. epts event processing technical society EP MetaPattern future versions? • Name & Alternative Names • Classification versus other references – EPTS RA and other references e.g. “EP in Action” book • Description may use revised EP Fns classification / ontology – Role, and Business Rule type specification / requirement • Structure – EPTS Glossary terms • Implementations may use EPL classification per EPTS Languages WG 34
  • 32. epts event processing technical society EP Pattern Implementations • Differ in Implementation type (Event Processing Language, Execution semantics, etc) • Sampled systems we cover: 1.TIBCO BusinessEvents Rete-production-rule-engine (with continuous query and event-expression options) 2.Oracle CEP Stream-processing engine 3.Prova Logic Programming Semantic CEP Rule Engine 4.IBM WebSphere Decision Server event-driven production-rule-engine & IBM Infosphere Streams stream processing engine 35
  • 33. epts event processing technical society Agenda • Introduction to the EPTS Functional Reference Architecture, and what it contains • Mapping Functions to Patterns: some sample Event Processing Patterns • EPTS member pattern implementation examples • Summary and future work on EP Patterns 36
  • 34. epts event processing technical society Some EP Patterns to review in more detail 1. Filter: filter out all events that have some property in their payload / data – E.g. customer purchases: filter out those with values < $100 2. Enrichment: add some information based on prior events – E.g. customer purchase: add the customer history to the purchase event 3. Aggregation: add prior events together for new information – E.g. customer purchase history: track the sum total of order values for a customer 4. Routing: based on the purchase type pass the event on to the appropriate service – E.g. customer purchase event: pass on to a provisioning service 37 based on the type of product / product classification
  • 35. epts event processing technical society Some EP Patterns to review in more detail 1. Filter: filter out all events that have some property in their payload / data – E.g. customer purchases: filter out those with values < $100 2. Enrichment: add some information based on prior events – E.g. customer purchase: add the customer history to the purchase event 3. Aggregation: add some information based on prior events – E.g. customer purchase history: track the sum total of order values for a customer 4. Routing: based on the purchase type pass the event on to the appropriate service – E.g. customer purchase event: pass on to a provisioning 38 service based on the type of product / product classification
  • 36. epts event processing technical society Filter Pattern: Classification (1) • EPTS Reference Architecture: – "During event preparation, a stream or list of events may be filtered on some payload or metadata information such that some subset is selected for further processing." • EPIA: – Filter (EPA or Event Processing Agent) performs filtering only and has no matching or derivation steps, so it does not transform the input event. 39
  • 37. epts event processing technical society Filter Pattern: Description (2) • Role / Explanation – Comparing some property of the event (an attribute or metadata) with some other value (from some other event, or data) • Associated Business Rule Specification – As a constraint: a selection (to be enforced during the processing of events): All <event entities> that have Filter <filter expression> event event f(event) must have <some status>. discarded event 40
  • 38. epts event processing technical society Filter Pattern: Structure (3) • EPTS Glossary comparison – A filter pattern can relate to several terms in the EPTS Glossary • can be specified in an event pattern (A template containing event templates, relational operators and variables...) • .. by an event pattern constraint (A Boolean condition that must be satisfied by the events observed in a system...) • .. as part of an event processing rule (A prescribed method for processing events.) • .. or as part of event stream processing (Computing on inputs that are event streams.) – 41
  • 39. epts event processing technical society Filter Pattern: Implementations (4) • TIBCO, Oracle, Prova and IBM implementations for this pattern Filter event event f(event) discarded event 42
  • 40. epts event processing technical society Filter Pattern: Implementation 1a TIBCO BusinessEvents: rules • General pattern: <declare> event event <if> event-boolean-expression Filter f(event) <then> update-event-with-fact event OR action-for-filtered-event Operation on • Effect: ignore filtered events event on operations on non-filtered events 43
  • 41. epts event processing technical society Filter Pattern: Implementation 1b TIBCO BusinessEvents: event removal rules • Implementation type: Rete-based rule-driven system • Alternate pattern: <declare> event event <if> NOT Filter NOT event-boolean-expression !f(event) <then> consume-event event • Effect: removed filtered events from system (and ALL subsequent operations!) 44
  • 42. epts event processing technical society Filter Pattern: Implementation 1c TIBCO BusinessEvents: State transitions • General pattern: <state models for concept C> event <route from State n to n+1> <on> event Filter <where> event-filter query(event) • Effect: change state for some concept State transition • Notes: this is a specialisation of the general rule pattern! 45
  • 43. epts event processing technical society Filter Pattern: Implementation 1d TIBCO BusinessEvents: Queries • General pattern: <select> output event <from> event <where> event-filter Filter query(event) <policy> notification Policy Event event subset • Effect: collect filtered events for event operations per some policy Operations on event subset • Note: the pattern matching framework on policy event has a similar notion 46 but a different surface syntax
  • 44. epts event processing technical society Filter Pattern: Implementation 2a Oracle CEP: queries • General pattern: event SELECT <property> FROM <STREAM> predicate WHERE <predicate-condition> event • Effect: discard events whose predicate Operation on evaluate to false event 47
  • 45. epts event processing technical society Filter Pattern: Implementation 2b Oracle CEP: model • Scenario: – Select only stocks from the stock tick stream whose symbols are „AAA‟. • Application Model 48
  • 46. epts event processing technical society Filter Pattern: Implementation 2c Oracle CEP: query Specify STREAM source SELECT * FROM StockTickStream Define predicate for WHERE symbol = „AAA‟ filtering 49
  • 47. epts event processing technical society Filter Pattern: Implementation 3a Prova: rules • Implementation type: backward reasoning logical filter • General pattern template: rcvMsg / rcvMult <Event Msg Pattern(s)> :- event <logical filter condition(s)>, … event ... . Event pattern + <logical derivation rule 1> :- logical rule filter <filter conditions>. Backward <logical derivation rule 2> :- Resoning Rule event <filter conditions>. chaning ... • Effect: Unify received events with Operation on event pattern definition and apply logical event 50 filter rules (derivation rules with rule chaining)
  • 48. epts event processing technical society Filter Pattern: Implementation 3b Prova: Messaging Reaction Rules in Prova • Send a message sendMsg(XID,Protocol,Agent,Performative,[Predicate|Args]|Context) • Receive a message rcvMsg(XID,Protocol,Agent,Performative,[Predicate|Args]|Context) • Receive multiple messages rcvMult(XID,Protocol,Agent,Performative,[Predicate|Args]|Context) Description: – XID is the conversation identifier – Protocol: protocol e.g. self, jms, esb etc. – Agent: denotes the target or sender of the message – Performative: pragmatic context, e.g. FIPA Agent Communication – [Predicate|Args] or Predicate(Arg1,..,Argn): Message payload
  • 49. epts event processing technical society Filter Pattern: Implementation 3c Prova: Example % Filter for stocks starting with „A“ and price > 100 rcvMult(SID,stream,“S&P500“, inform, tick(S,P,T)) :- S = "A.*", P > 100, sendMsg(SID2,esb,"epa1", inform, happens(tick(S,P),T). Example with rule chaining % Filter for stocks starting with „A“ and price > 100 rcvMult(SID,stream,“S&P500“, inform, tick(S,P,T)) :- filter(S,P), sendMsg(SID2,esb,"epa1", inform, happens(tick(S,P),T). filter(Symbol,Price) :- Price > 100, Symbol = "A.*".
  • 50. epts event processing technical society Filter Pattern: Implementation 3d Prova: Semantic Event Processing Example Semantic Query Filer: Stocks of companies, which have production facilities in Europe and produce products out of metal and Have more than 10,000 employees. Event Stream {(Name, “OPEL”)(Price, 45)(Volume, 2000)(Time, 1) } {(Name, “SAP”)(Price, 65)(Volume, 1000) (Time, 2)} {(OPEL, is_a, car_manufacturer), (car_manufacturer, build, Cars), (Cars, are_build_from, Metall), Semantic (OPEL, hat_production_facilities_in, Germany), Knowledge Base (Germany, is_in, Europe) (OPEL, is_a, Major_corporation), (Major_corporation, have, over_10,000_employees)} rcvMult(SID,stream,“S&P500“, inform, tick(Name^^car:Major_corporation,P^^currency:Dollar, T^^time:Timepoint)) :- … <semantic filter inference> .
  • 51. epts event processing technical society Filter Pattern: Implementation 4a IBM WebSphere Decision Server: Business Events component event event • Example: Define a filter to only process events with a Filter purchase value greater than £100 event Operation on • Effect: Only process events event which match the filter 55
  • 52. epts event processing technical society Filter Pattern: Implementation 4b IBM InfoSphere Streams: Streams Processing Language • Example: composite Main { graph stream<rstring name, uint32 age> Beat = Beacon() {} stream<rstring name, uint32 age> Youngs = Filter(Beat) { param filter : age < 30u; } (stream<Beat> Younger; stream<Beat> Older) = Filter(Beat) { param filter : age < 30u; } } • Effect: filter out into streams of older and younger ages 56
  • 53. epts event processing technical society Some EP Patterns to review in more detail 1. Filter: filter out all events that have some property in their payload / data – E.g. customer purchases: filter out those with values < $100 2. Enrichment: add some information based on prior events – E.g. customer purchase: add the customer history to the purchase event 3. Aggregation: add some information based on prior events – E.g. customer purchase history: track the sum total of order values for a customer 4. Routing: based on the purchase type pass the event on to the appropriate service – E.g. customer purchase event: pass on to a provisioning 57 service based on the type of product / product classification
  • 54. epts event processing technical society Enrichment Pattern: Classification (1) • Alternative Names: – Event Annotation, Semantic Event • EPTS Reference Architecture: – "During event preparation, events may be "enriched" through knowledge gained through previous events or data. " • EPIA: – Enrich (EPA): a translate EPA that takes a single input event, uses it to query data from a global state element, and creates a derived event which includes the attributes from the original event, possibly with modified values... 58
  • 55. epts event processing technical society Enrichment Pattern: Description (2) • Role / Explanation – Updating some attribute or metadata of the event with some other values, possibly computed, from some other event or data • Associated Business Rule Specification – a selection (to be enforced during the processing of events): All <event entities> that are also <enriched data expression> Other data must have <some status>. or event Enrichment event event f(event) 59
  • 56. epts event processing technical society Enrichment Pattern: Structure (3) • EPTS Glossary comparison – Enrichment can be specified in an event pattern (pattern definition containing additional meta data, semantic knowledge) – .. as part of an event processing rule (A method for enriching events during processing, e.g., by analysing the meta data and querying external data sources or inferencing semantic knowledge bases) – .. or as part of event stream processing (additional additional information to event stream data, e.g. by querying external data sources) 60
  • 57. epts event processing technical society Enrichment Pattern: Implementations (4) • TIBCO, Oracle, Prova and IBM implementations for this pattern Other data or event Enrichment event event f(event) 61
  • 58. epts event processing technical society Enrichment Pattern: Implementation 1 TIBCO BusinessEvents: Rules • General pattern: Enrich <declare> event, data data event <if> event-data-match Identify enrichment data <then> update-event-with-data Enrich Operation • Effect: find appropriate data for on event events and apply enrichment operations on them Event 62
  • 59. epts event processing technical society Enrichment Pattern: Implementation 2a Oracle CEP: queries • General pattern template: event SELECT <property> FROM <STREAM>[WIN],<RELATION> join-condition WHERE <join-condition> event • Effect: enrich events from stream with contextual data from relation where condition applies. Operation on event 63
  • 60. epts event processing technical society Enrichment Pattern: Implementation 2b Oracle CEP: model • Scenario: – For every stock event from stock tick stream whose bid is greater than 5.0, find its company‟s location. • Application Model 64
  • 61. epts event processing technical society Enrichment Pattern: Implementation 2c Oracle CEP: queries No need for window SELECT event.symbol, location operator on a „relation‟ FROM StockTickStream [NOW] AS event, StockTable AS data WHERE event.symbol = data.symbol AND event.lastBid > 5.0 65
  • 62. epts event processing technical society Enrichment Pattern: Implementation 2d Oracle CEP: table metaphor StockTable Symbol Full-name Location AAA The AAA company San Francisco BBB The BBB company San Jose Time Input (StockTickStream) Output (QUERY) 1 {“AAA”, 10.0, 10.5} {“AAA”, “San…”} 2 {“BBB”, 11.0, 12.5} {“BBB”, “San Jose”} 3 {“AAA”, 4.0, 4.5} 66
  • 63. epts event processing technical society Enrichment Pattern: Implementation 3a Prova: rules • General pattern template: rcvMsg / rcvMult <Event Msg Pattern(s)> :- event <query built-in(s)>, <enrichment condition(s)> External ... . Query data <logical enrichment rule 1> :- <enrichment operations>. Enrichment rules • Effect: query data from external source event* and enrich the event Operation on enriched event 67
  • 64. epts event processing technical society Enrichment Pattern: Implementation 3b Prova: External Data and Object Integration • Java ..., L=java.util.ArrayList(),, ..., java.lang.String(“Hello”) • File Input / Output ..., fopen(File,Reader), ... • XML (DOM) document(DomTree,DocumentReader) :- XML(DocumenReader), • SQL …,sql_select(DB,cla,[pdb_id,“1alx“],[px,Domain]). • RDF ...,rdf(http://...,"rdfs",Subject,"rdf_type","gene1_Gene"), • XQuery ..., XQuery = ' for $name in StatisticsURL//Author[0]/@name/text() return $name', xquery_select(XQuery,name(ExpertName)), • SPARQL ...,sparql_select(SparqlQuery,name(Name),class(Class), definition(Def)),
  • 65. epts event processing technical society Enrichment Pattern: Implementation 3c Prova: Example with SPARQL Query % Filter for car manufacturer stocks and enrich the stock tick event with data from Wikipedia (DBPedia) about the manufacturer and the luxury cars rcvMult(SID,stream,“S&P500“, inform, tick(S,P,T)) :- carManufacturer(S,Man), % filter car manufacturers findall([luxuryCar|Data],luxuryCar(Man,Name,Car),Data), % query EnrichedData = [S,Data], % enrich with additional data sendMsg(SID2,esb,"epa1", inform, happens(tick(EnrichedData,P),T). % rule implementing the query on DBPedia using SPARQL query luxuryCar(Manufacturer,Name,Car) :- Query="SELECT ?manufacturer ?name ?car % SPARQL RDF Query WHERE {?car <http://purl.org/dc/terms/subject> <http://dbpedia.org/resource/Category:Luxury_vehicles> . ?car foaf:name ?name . ?car dbo:manufacturer ?man . ?man foaf:name ?manufacturer. } ORDER by ?manufacturer ?name“, sparql_select(Query,manufacturer(Manufacturer),name(Name),car(Car)).
  • 66. epts event processing technical society Enrichment Pattern: Implementation 4 IBM WebSphere Decision Server: Business Events component • Example: Define a Mapped Key expression, where a field in the event is used to look up objects in a database table event Mapped Key look up Enriched event • Effect: required enrichment data is obtained 70 from the data base and added to event
  • 67. epts event processing technical society Some EP Patterns to review in more detail 1. Filter: filter out all events that have some property in their payload / data – E.g. customer purchases: filter out those with values < $100 2. Enrichment: add some information based on prior events – E.g. customer purchase: add the customer history to the purchase event 3. Aggregation: add some information based on prior events – E.g. customer purchase history: track the sum total of order values for a customer 4. Routing: based on the purchase type pass the event on to the appropriate service – E.g. customer purchase event: pass on to a provisioning 71 service based on the type of product / product classification
  • 68. epts event processing technical society Aggregation Pattern: Classification (1) • Alternative Names: – Event Summarization, Composite event, Complex event • EPTS Reference Architecture: – "During complex event detection, combining events to provide new or useful information, such as trend information and event statistics. Similar to event consolidation " • EPIA: – Aggregate (EPA): a transformation EPA that takes as input a collection of events and creates a single derived event by applying a function over the input events. 72
  • 69. epts event processing technical society Aggregation Pattern: Description (2) • Role / Explanation – Aggregation is used to describe several events as a single composite event, generally summarizing a metric of a set of component events. – For example: generate a (composite) event that contains the medium price of a stock over a 10 minute stream of events. • Associated Business Rule Specification – a summarisation (to be enforced during event processing): The <aggregation fn> of <event entities> that are <selection constraint> must have Aggregation <some constraint>. event event event f(events) 73 event
  • 70. epts event processing technical society Aggregation Pattern: Structure (3) • EPTS Glossary comparison – The term aggregate event is sometimes used for some forms of composite or derived event. – Composite event: a derived, complex event • is created by combining base events using a specific set of event constructors such as disjunction, conjunction, sequence, etc. • always includes the base (member) events from which it is derived. – Derived event (/synthesized event): an event that is generated as a result of applying a method or process to one or more other events. 74
  • 71. epts event processing technical society Aggregation Pattern: Implementations (4) • TIBCO, Oracle, Prova and IBM implementations for this pattern Aggregation event event event f(events) event 75
  • 72. epts event processing technical society Aggregation Pattern: Implementation 1 TIBCO BusinessEvents: Rules • General pattern: Aggregation <declare> event, object event aggregation-object Identify <if> event for event-member-of-aggregation aggregation <then> update-aggregation-object Aggregate with event-data Operation on event • Effect: construct aggregation Updated (event) object for each incoming event Aggregation 76 Object
  • 73. epts event processing technical society Aggregation Pattern: Implementation 2a Oracle CEP: queries • General pattern template: Aggregation object event SELECT <summary-property> FROM <STREAM>[WIN] Identify WHERE <predicate> event for GROUP BY <aggregation-identity> aggregation • Effect: summarizes the properties of Aggregate several simple event into a new Operation complex event on event Updated Aggregation 77 Object
  • 74. epts event processing technical society Aggregation Pattern: Implementation 2b Oracle CEP: model • Scenario: – Output the average bid and ask price of a stock in the last 10 seconds. • Application Model 78
  • 75. epts event processing technical society Aggregation Pattern: Implementation 2c Oracle CEP: queries SELECT symbol, AVG(bid), AVG(ask) FROM StockTickStream [RANGE 10 SECONDS] GROUP BY symbol 79
  • 76. epts event processing technical society Aggregation Pattern: Implementation 2d Oracle CEP: table metaphor Time Input Output 1s {“AAA”, 10.0, 12.0} {“AAA”, 10.0, 12.0} 4s {“AAA”, 12.0, 14.0} {“AAA”, 11.0, 13.0} {“AAA”, 11.0, 13}, 9s {“BBB”, 4.0, 5.0} {“BBB”, 4.0, 5.0} 15s {“BBB”, 8.0, 10.0} {“BBB”, 6.0, 7.5} > 10s 20s {“BBB”, 8.0, 10.0} 80
  • 77. epts event processing technical society Aggregation Pattern: Implementation 3a Prova: rules Aggregation object event • General pattern template: <rule_head> :- Check & Update <Create Aggregator>, Timer / Size Count @group(<reaction group>) @timer|size(Start,Interval,Aggregator) Identify event rcvMsg <EventPattern> [<Aggregate Operation>]. aggregation for or <rule_head> :- Aggregate @or(<reaction group>) Operation rcvMsg <Aggregator>, on event ... <consume Aggregator>. • Effect: Repeated incremental Updated Aggregation Consume Aggregation aggregations over new events Object Object 81
  • 78. epts event processing technical society Aggregation Pattern: Implementation 3b Prova: Example with Time Counter % This reaction operates indefinitely. When the timer elapses (after 25 ms), the groupby map Counter is sent as part of the aggregation event and consumed in or group, and the timer is reset back to the second argument of @timer. groupby_rate() :- Counter = ws.prova.eventing.MapCounter(), % Aggr. Obj. @group(g1) @timer(25,25,Counter) % timer every 25 ms rcvMsg(XID,stream,From,inform,tick(S,P,T)) % event [IM=T,Counter.incrementAt(IM)]. % aggr. operation groupby_rate() :- % receive the aggregation counter in the or reaction @or(g1) rcvMsg(XID,self,From,or,[Counter]), ... <consume the Counter aggreation object>.
  • 79. epts event processing technical society Aggregation Pattern: Implementation 4a IBM WebSphere Decision Server: Business Events component • General Pattern: event event Define how the events should be accumulated Add to array ... then when processing the events, use the Intermediate relevant Object Array aggregator • Effect of the example: sum up the value of a Operation customer‟s purchases over a week on aggregate 83
  • 80. epts event processing technical society Aggregation Pattern: Implementation 4b IBM InfoSphere Streams: Streams Processing Language • Example: // sliding window stream<Beat, tuple<int32 maxSalary, uint32 ageOfMaxSalary> > Agg4 = Aggregate(Beat) { window Beat : sliding, time(10.5), count(10); output Agg4 : maxSalary = Max(salary), ageOfMaxSalary = ArgMax(salary, age); } • Effect of example: Aggregate the salary values in the stream within a time-based window. 84
  • 81. epts event processing technical society Some EP Patterns to review in more detail 1. Filter: filter out all events that have some property in their payload / data – E.g. customer purchases: filter out those with values < $100 2. Enrichment: add some information based on prior events – E.g. customer purchase: add the customer history to the purchase event 3. Aggregation: add some information based on prior events – E.g. customer purchase history: track the sum total of order values for a customer 4. Routing: based on the purchase type pass the event on to the appropriate service – E.g. customer purchase event: pass on to a provisioning 85 service based on the type of product / product classification
  • 82. epts event processing technical society Routing Pattern: Classification (1) • Alternative Names: – Event Summarization, Composite event, Complex event • EPTS Reference Architecture: – "During event reaction, event routing is the process by which an event is redirected to some process, computation element, or other event sink. " • EPIA: no direct analogy, possibly maps to split / compose / project – Project (EPA): a translate EPA that takes an input event and creates a single derived event containing a subset of the attributes of the input event. 86
  • 83. epts event processing technical society Routing Pattern: Description (2) • Role / Explanation – Routing is used to describe the process of adding one or more new destinations to an event. – For example: route an input event to the appropriate specialist agent for that event type. • Associated Business Rule Specification – a constraint (to be enforced during event processing): The <event> that Other data satisfies <selection constraint> or event must be assigned to <destination>. Routing event event dest = f(event+data) 87
  • 84. epts event processing technical society Routing Pattern: Structure (3) • EPTS Glossary comparison – Routing (a process on events) is not defined, but is related to associating an event to an event sink. – Event sink (event consumer) is an entity that receives events. • Examples: • ƒ Software module • ƒ Database • ƒ Dashboard • ƒ Person 88
  • 85. epts event processing technical society Routing Pattern: Implementations (4) • TIBCO, Oracle, Prova and IBM implementations for this pattern Other data or event Routing event event dest = f(event+data) 89
  • 86. epts event processing technical society Routing Pattern: Implementation 1 TIBCO BusinessEvents: Rules • General pattern: Routing <declare> event, data event routing-data Identify <if> event for event-matches-routing-data routing <then> Create routed create-new-routed-event and event send-new-routed-event event • Effect: match events to routing rule and construct / send new event 90
  • 87. epts event processing technical society Routing Pattern: Implementation 2a Oracle CEP: query pattern • General pattern template: Routing data event Route 1: <filtering pattern> query Route 2: <filtering pattern> query Identify Route n: <filtering pattern> query event for • Effect: Different predicates routing associated to different queries evaluate and select appropriate Select destination destinations event 91
  • 88. epts event processing technical society Routing Pattern: Implementation 2b Oracle CEP: model • Scenario: – Output stocks whose symbols start with „a-m‟ to destination 1, and whose symbols start with „n-z‟ to destination 2. • Application Model 92
  • 89. epts event processing technical society Routing Pattern: Implementation 2c Oracle CEP: queries Destination-1: SELECT * FROM StockTickStream WHERE symbol.matches(“^[a-m]”) Destination-2: SELECT * FROM StockTickStream WHERE symbol.matches(“^[n-z]”) 93
  • 90. epts event processing technical society Routing Pattern: Implementation 3 event Prova: rules • General pattern template: Identify event for rcvMsg <Event Msg Pattern> :- routing <routing decisions>, sendMsg <route event >. Routing Routing Decisions data <routing decision rule 1> :- <decision conditions>. Create routed <routing decision rule 2> :- event <decision conditions>. ... Route event event • Effect: Events are routed according to the routing decision 94 rules
  • 91. epts event processing technical society Routing Pattern: Implementation 3b Prova: Example with Agent (Sub-) Conversations rcvMsg(XID,esb,From,query-ref,buy(Product) :- routeTo(Agent,Product), % derive processing agent % send order to Agent in new subconversation SID2 sendMsg(SID2,esb,Agent,query-ref,order(From, Product)), % receive confirmation from Agent for Product order rcvMsg(SID2,esb,Agent,inform-ref,oder(From, Product)). % route to event processing agent 1 if Product is luxury routeTo(epa1,Product) :- luxury(Product). % route to epa 2 if Product is regular routeTo(epa2,Product) :- regular(Product). % a Product is luxury if the Product has a value over … luxury(Product) :- price(Product,Value), Value >= 10000. % a Product is regular if the Product ha a value below
  • 92. epts event processing technical society Routing Pattern: Implementation 4 IBM: Business Events component of WebSphere Decision Server Routing event • Example: Define an interaction set, information which will send the request to supplier A, when the purchase indicates supply from Supplier A Identify • event routing event • Effect of example: Events are routed as required Consume 96 event
  • 93. epts event processing technical society Agenda • Introduction to the EPTS Functional Reference Architecture, and what it contains • Mapping Functions to Patterns: some sample Event Processing Patterns • EPTS member pattern implementation examples • Summary and future work on EP Patterns 97
  • 94. epts event processing technical society Another Pattern Type Other patterns may be required for implementation, such as handling produced-consumer interactions
  • 95. epts event processing technical society Application-Time Pattern • Problem: – Need to use application‟s view of time, instead of CPU wall-clock. – This is particularly useful when events originate from different machines and need some way of synchronizing. • Scenario: – Again, consider bid and ask stream… – However, bid and ask requests are time-stamped on trader at the time that the order is placed. 99
  • 96. epts event processing technical society Application-Time Pattern • Scenario: – Seller places two ask requests respectively at time 8:00:00 and 8:00:12 – Buyer places one bid request at time 8:00:11 100
  • 97. epts event processing technical society Application-Time Pattern • Scenario: – Remember that we want to correlate using a 10 seconds window, as anything older is stale… – Hence first event from seller should not be considered, and we should correlate the ask price of 11.0 with the bid price of 9.5 101
  • 98. epts event processing technical society Application-Time Pattern • Scenario: – However, the reality is that the first two events could arrive together in a burst in the exchange, and the third could be delayed in the cloud… 102
  • 99. epts event processing technical society Application-Time Pattern • Scenario: – In this case, bid price of 9.5 would correlate to ask price of 10.0 and the exchange would loose money as the spread is lower… 103
  • 100. epts event processing technical society Application Time-stamped Pattern: Implementation 1 Oracle CEP • Solution: – The query does not change… – However STREAMS must be configured to use application time-stamps based upon some event property, instead of having events being system- timestamped… 104
  • 101. epts event processing technical society Application Time-stamped Pattern: Implementation 2 IBM Business Monitor • Example: Define a correlation expression „Order Placed‟ „Order Complete‟ event event • Effect of example: The duration of a customer order can be obtained from the timestamps on start and end events Correlate Calculate Duration 105
  • 102. epts event processing technical society Application Time Pattern: Implementation 3 Prova: rules • General pattern template: Event rcvMsg <Event Msg Pattern(s)> :- <add application timestamp to event>, sendMsg <Adjusted Event>. adjust event • Effect: Any timestamp can be adjusted (for time computations) Updated Event 106
  • 103. epts event processing technical society Application Time-stamped Pattern: Implementation 4 TIBCO BusinessEvents: PreProcessor Pattern • General Pattern: adjust Event metadata (e.g. Timestamp) in preprocessor for that event Event – events are generally non-monotonic! – PreProcessor applies regardless of engine used! Preprocess: • Effect: adjust event Any custom timestamp can be adjusted (for time computations) Note: If the problem is “out of order” events Updated Event (rather than synchronizing clocks) a different pattern may be better! 107
  • 104. epts event processing technical society Use Cases Illustrating using some of the patterns
  • 105. epts event processing technical society IBM Use Case: Utilizing Smart Meters Large energy company improved customer responsiveness using event processing Use Case • Utilized events (and lack of events) from smart meters to detect service outages and disruptions • Empowered operations team to define outage detection and response patterns without coding or scripting Smarter Business Outcomes • Automated detection and response to service Challenge: Reliance on customer calls outages to detect and respond to outages and reduce time to recovery while providing • Improved customer satisfaction by delivery of more information to the customer estimated time to recovery alerts based on customer preferences
  • 106. epts event processing technical society Energy Company Leveraging Smart Meters to Improve Customer Responsiveness Customer Business Data Events 2 1 3 Data Collection Engine (DCE) WebSphere Business Events 1 4 Tivoli OMNIbus, Network Manager, and Impact Power Outage Disruption Management Customer Care System (OMS) Manager(s) with IBM Business Monitor Location, Estimated Duration of Disruption Alert and Assist Impacted Customers 110
  • 107. epts event processing technical society Oracle Use-case: Capital Markets • Low-Latency Trading – Algorithm Trading • Trade Matching – High throughput – In-flight fraud • Position Capture and Aggregation • Risk and Analytics – Pre/post trade analytics – On-demand risk management 111
  • 108. epts event processing technical society TIBCO Use Case: Application Service Gateway Telco high performance service policies using event processing Excellent service quality Mobile Services needed to avoid difficult to monitor customer churn end to end Auto detect and fixing of issues Service quality affects for better take-up of new services 112 user service
  • 109. epts event processing technical society Service Gateway architecture exploits event processing • Standardized channel for monitoring subscribers • Standardized access to services (e.g. Facebook, Google, etc) • Consolidation of existing multiple gateways • Allows analysis of service usage, subscriber behavior, service performance, marketing campaign success, etc • Creates events for revenue generation and reconciliation • Handles Common Telco Notifications - change-of-device, contract, service- allowing UpSell/Cross sell etc
  • 110. epts event processing technical society Service Gateway architecture exploits event processing • Standardized channel for monitoring subscribers • Standardized access to services (e.g. Facebook, Google, etc) Security Pattern • Consolidation of existing multiple gateways Decision / Policy • Allows Pattern analysis of service usage, subscriber behavior, service performance, marketing campaign success, etc Routing Pattern • Creates events for revenue generation and reconciliation Transformation • Handles Common Telco Patterns Notifications - change-of-device, contract, service- allowing UpSell/Cross sell etc
  • 111. epts event processing technical society Agenda • Introduction to the EPTS Functional Reference Architecture, and what it contains • Mapping Functions to Patterns: some sample Event Processing Patterns • EPTS member pattern implementation examples • Summary and future work on EP Patterns 115
  • 112. epts event processing technical society Summary: Realization Patterns / Idioms Event Reaction Assessment, Routing, Prediction, Discovery, Learning State Management 0..* Complex Event Detection Consolidation, Composition, Aggregation 0..* Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification 0..* Event Preparation Identification, Selection, Filtering, Monitoring, Enrichment 0..* Event Production Event Consumption Application Time Dashboard, Apps, Publication, Retrieval External Reaction
  • 113. epts event processing technical society Categorization of Event Processing Patterns Source:Paschke, A.: Design Patterns for Complex Event Processing, DEBS'08, Rome, Italy, 2008 http://arxiv.org/abs/0806.1100v1
  • 114. epts event processing technical society Categorization of Patterns • Categorization according to Good and Bad Solutions – CEP Patterns – CEP Anti-Patterns • Categorization according to the Abstraction Level – Guidelines and Best Practices – Management patterns – Architecture patterns – Design patterns – Mapping patterns – Idioms / Realization patterns – Smells / Refactoring patterns • Categorization according to the Intended Goal – Adoption patterns – Business patterns – Integration patterns – Composite patterns: – … • Categorization according to the Management Level – Strategic patterns – Tactical patterns Source:Paschke, A.: A Semantic Design Pattern Language for Complex Event Processing, – Operational patterns Intelligent Event Processing, Papers from the 2009 AAAI Spring Symposium (2009) , p. 54-60. http://www.aaai.org/Papers/Symposia/Spring/2009/SS-09-05/SS09-05-010.pdf
  • 115. epts event processing technical society Next Steps • Continue evolution of EP Reference Architecture – Eg Logical Architecture vs Systems Description • Evolve the EP Pattern <<metapattern>> to be more useful for developers and tools • Expand coverage to all Functions as Patterns, + extend Functions as necessary • Planned Outputs: – EPTS-RA Description white paper – Wiley Book "Pattern Oriented Software Architecture: Architectures, Models and Patterns for Event Processing" (already in preparation) – Input for Event Processing Metamodels and associated standards
  • 116. epts event processing technical society Summary • Reference Architecture provides a common set of EP Functions that may be mapped to other EP ontologies / taxonomies • Provides the basis for discussing EP Patterns • EP Patterns can be semantic, design or runtime focused • Further evolution of EP Pattern and examples is likely! 120
  • 117. epts event processing technical society Thank you ! Acknowledgment to the Event Processing Technical Society Reference Architecture working group members 121
  • 118. epts event processing technical society References • EPTS Glossary http://www.ep-ts.com/component/option,com_docman/task,doc_download/gid,66/Itemid,84/