SlideShare a Scribd company logo
1 of 115
epts
                                                   event processing technical society




                                      Event Processing
                                      Language Tutorial
François Bry, Michael Eckert, Opher Etzion, Adrian Paschke, Jon Riecke
on behalf of the epts languages analysis working group
epts
    event processing technical society




TUTORIAL OUTLINE

       •    Introduction                          Opher Etzion
       •    stream processing languages           Jon Riecke
       •    rule oriented languages               Adrian Paschke   Languages
                                                                   Styles
       Break
       •    Agent oriented languages              Opher Etzion
       •    Temporal semantics                    Jon Riecke
                                                                   Advanced
       •    Integrated development environments   Jon Riecke
                                                                   Topics
       •    Formal approaches                     Adrian Paschke
       •    Conclusion                            Opher Etzion




2
epts
    event processing technical society




TUTORIAL OUTLINE

       •    Introduction                          Opher Etzion
       •    stream processing languages           Jon Riecke
       •    rule oriented languages               Adrian Paschke   Languages
                                                                   Styles
       Break
       •    Agent oriented languages              Opher Etzion
       •    Temporal semantics                    Jon Riecke
                                                                   Advanced
       •    Integrated development environments   Jon Riecke
                                                                   Topics
       •    Formal approaches                     Adrian Paschke
       •    Conclusion                            Opher Etzion




3
epts
     event processing technical society


What is it all about?
                                                                                   Filter, transform, enrich, route,
                                                                                    Detect patterns, derive events

                                                          Event Processing



              Event                                                                                 Event
             Producer                                                                             Consumer


         Generate and emit                                                                       Consume and react
         events                                        perform operations on                     to either raw or
                                                       events                                    complex events

                                          Event processing has
                                          gained a lot of interest in
                                          recent years; according to
                                          analysts it is the fastest
                                          growing segment of
                                          middleware software

                                                                             In this tutorial we are          Pattern
                                                                             exploring the language           Match
                                                                             issue – how the various
                                                                             event processing
                                                                             functions are expressed?

 4
epts
      event processing technical society

An Observation                                The Babylon Tower symbolizes the tendency
                                              Of humanity to talk in multiple languages.




    The Event Processing area is no different: most languages in the industry really follow
    the hammer and nails syndrome – and extended existing approaches
    • imperative script language
    • SQL extensions
    • Extension of inference rule language



                        It does not seem that we’ll succeed to settle
                        In the near future around a single programming style



                               The epts language analysis workgroup is aimed to understand the various styles
                               And extract common functions that can be used to define what is an event
                               processing language; this tutorial is an interim report



5
epts
       event processing technical society



    Existing Styles for EP languages (samples)
                                                  Inference        TIBCO
                                                    Rules
     Prova                                  WBE                               XChangeEQ
                                                    ECA
                                                   Rules                         RuleCore
                                                                   AMiT
     Starview                                      Agent                   Agent Logic
                                                  Oriented
                    EventZero                                      Spade
                                                    SQL
Aleri                                             extension                  Streambase

    Coral8
                                                    State           Oracle
                                                   oriented
                       Esper

                                                    Imperative/
    Netcool Impact                                  Script Based             Apama
    * - if we add simple and mediated event processing the picture is even more diversified
6
epts
    event processing technical society




Dimensions to describe languages

       • Events Data and Meta-data
       • State
       • Computation/execution model
       • Programming model




7
epts
    event processing technical society




The event Data and meta-data dimension

       • Event structures supported (structured, semi-
            structured, unstructured)
       • Data types supported for event’s payload
       • Event identifiers/identities
       • Event time-stamps and ordering
       • Event relations
       • Other meta-data entities in the language


8
epts
    event processing technical society




The state dimension

       •      State of events / data
            – Events and other data in states
            – Life-cycle, duration policies
            – Explicit visibility and manipulation of states
       •      State of execution
            – Context-related grouping (e.g. time windows)
            – Explicit visibility into state of execution




9
epts
     event processing technical society




The execution model dimension

        • Execution timing – immediate or deferred ?
        • Guarantees: real-time, determinism, latency ?
        • Time point or time interval semantics
        • Concurrency – implicit / explicit
        • Event at a time / set at a time processing
        • Explicit event flow



10
epts
     event processing technical society




The programming model dimension

        •      Programming style
        •      Abstractions
        •      Pattern matching
        •      Enrichment capabilities
        •      Transformation capabilities
        •      Filter capabilities




11
epts
     event processing technical society




TUTORIAL OUTLINE

        Introduction                              Opher Etzion
             stream processing languages           Jon Riecke
        •    rule oriented languages               Adrian Paschke   Languages
                                                                    Styles
        Break
        •    Agent oriented languages              Opher Etzion
        •    Temporal semantics                    Jon Riecke
                                                                    Advanced
        •    Integrated development environments   Jon Riecke
                                                                    Topics
        •    Formal approaches                     Adrian Paschke
        •    Conclusion                            Opher Etzion




12
epts
     event processing technical society




Hallmarks of Stream Processing

        •    Events are processed in a directed graph
             • Nodes = computational elements
             • Edges = communication of events between nodes
        •    Languages inspired by SQL/relational algebra (though not
             all use an extended SQL as the language)
             • Difference: write the query first
             • Results of query are updated as data flows in
        •    Academic examples: STREAM, Aurora, Cayuga, …
        •    Commercial examples: Aleri, Coral8, Esper/Nesper,
             StreamBase, System S, …




13
epts
     event processing technical society




Some pictures




14
epts
     event processing technical society




Some pictures




15
epts
     event processing technical society




Some pictures




16
epts
     event processing technical society




Historical Roots

        • Relational Databases
             • Triggers: change table when another table changes
             • Materialized view maintenance: calculate small set of
                  changes to view given set of changes to base tables
        • Signal processing
             • Control and Audio: Ptolemy, LabVIEW, …
             • Music and video: OpenMusic, jMax, Max, vvvv,



17
epts
     event processing technical society




Historical Roots: Static Dataflow Architecture




        • Developed by Jack Dennis in 1960s
        • Each arc carries one token
        • Node fires when all data available on all arcs
18
epts
     event processing technical society




Historical Roots: Dynamic Dataflow Architecture




        • Matching section in front of each node
        • Tokens are tagged by invocation instance; when all
             tokens with same tag are ready, node fires
        • Allows more parallelism than static model
19
epts
     event processing technical society




Kahn Dataflow Networks




        • Non-blocking write/blocking read queues
        • Deterministic nodes implies deterministic network
20
epts
     event processing technical society




Stream Processing versus
   Signal Processing and Dataflow Architecture

        • Events carry more information than simple data:
             records instead of, e.g., decimal numbers
        • Slightly slower data rates (up to millions of events
             per second)
        • Publish/subscribe model of input/output




21
epts
     event processing technical society




Dimension Commonalities in Stream Processing

        •    Events: Non-mutable records (fields with values)
             Values have scalar types (integer, float, string, blob, xml)
        •    Metadata: describes streams and types of records
        •    State: Unprocessed and processed events, and internal
             state of operations (e.g., aggregations)
        •    Computational model: records traverse the dataflow graph
        •    Programming model: extensions of SQL/relational algebra




22
epts
     event processing technical society




Dimension Differences

        • Events: small differences in scalar types
        • State
                  •     User-defined data structures (arrays, maps, …)?
                  •     Disk-based persistence? What is persisted?
        • Programming model
                  •     Visual programming or text-based?
                  •     Embedded in larger language, e.g., Java?
                  •     Smaller language embedded in it?



23
epts
     event processing technical society




Dimension Differences

        • Computational model
             • Cycles in dataflow graph?
             • Deterministic?
             • Consistent? What form of consistency?
             • Parallel or concurrent execution?
             • Distributed computation? Distributed data?
             • Synchronous or asynchronous communication?



24
epts
     event processing technical society




Aleri Streaming Platform Dataflow Networks




        • One queue per node (instead of per edge)
        • Nodes store records in table
        • Events (insert/update/delete) change the table
        • Nodes run in separate threads
        • Optional persistence with roll-forward recovery
25
epts
     event processing technical society




Aleri Streaming Platform Dataflow Networks




        • Computational model
                  •     Events may arrive in different orders, and may cause
                        different output events
                  •     Correctness: standard operators (join, compute, …)
                        satisfy “eventual consistency”: the sequence of
                        insert/update/deletes will produce the same table, no
                        matter how the events flow through the graph
26
epts
     event processing technical society




Aleri Streaming Platform Dataflow Networks
                                                    B



                                          A                       C

                  •     Node A sends insert:[Id=3, Symbol=IBM, Price=45.0] Node B sends update:
                        [Symbol=IBM, AvgPrice=43.40]
                  •     Different arrival orders can cause different events
                            •    (A then B)
                            •            insert: [Id=3, Symbol=IBM, Price=45.0, AvgPrice=42.00]
                            •            update: [Id=3, Symbol=IBM, Price=45.0, AvgPrice=43.40]
                            •    (B then A)
                            •            insert: [Id=3, Symbol=IBM, Price=45.0, AvgPrice=43.40]



27
epts
     event processing technical society




Coral8 Dataflow Networks




        • No queues; synchronization buffer in front
        • Two kinds of nodes:
                  •     Stream nodes process records but don’t store them
                  •     Window nodes process and store records

        • Optional persistence for unprocessed events, records
28           in windows, aggregate state
epts
     event processing technical society




Coral8 Dataflow Networks




        • Computational model (simplified)
                  •     External data flows into synchronization buffer
                  •     Events carry a timestamp (point in time, microsecond
                        granularity): set on arrival, or by source
                  •     Events with same timestamp are fed into the network in
                        one step, and are processed until network quiesces
29
epts
     event processing technical society




Coral8 Dataflow Networks




        • Computational model
                  •     Deterministic: CCL programs produce the same output
                        events given the same input
                  •     Priorities assigned to nodes in graph to maintain
                        determinacy

30
epts
     event processing technical society




Coral8 Dataflow Networks




        • Computational model: concurrent/distributed
                  •     Project = group of CCL streams, windows, queries
                        (code for calculating new events from old)
                  •     Projects can send/receive events to/from other projects
                  •     Each project has a thread (there are other threads too)

31
                  •     Projects can be distributed across machines
epts
     event processing technical society




Examples in CCL: Simple Market Data

        • Schema: type of records
             – CREATE SCHEMA Trades_t
             – (Symbol STRING, Qty INTEGER, Price FLOAT);
             – CREATE SCHEMA Book_t
             – (Symbol STRING, Qty INTEGER, BuyPrice
               FLOAT);

        • Other scalar types: BOOLEAN, TIMESTAMP,
             INTERVAL, XML, BLOB

        • CCL streams: INPUT, OUTPUT, local
             – CREATE INPUT STREAM Trades_s SCHEMA Trades_t;
             – CREATE INPUT STREAM Book_s SCHEMA Book_t;


32
epts
     event processing technical society




CCL Windows

        • Windows are like streams, but store records
             – CREATE WINDOW Book_w SCHEMA Book_t KEEP ALL;
             – INSERT INTO Book_w
             –   SELECT * FROM Book_s;

        • Sample of KEEP policies (there are others):
             –    KEEP           LAST PER Id
             –    KEEP           3 MINUTES
             –    KEEP           EVERY 3 MINUTES
             –    KEEP           UNTIL (”MON 17:00:00”)
             –    KEEP           10 ROWS
             –    KEEP           LAST ROW
             –    KEEP           10 ROWS PER Symbol




33
epts
     event processing technical society




CCL Queries: SQL-based syntax

        •    Aggregation
             – CREATE STREAM Vwap_s SCHEMA (Symbol STRING, Vwap FLOAT);
             – INSERT INTO Vwap_s
             –   SELECT Symbol, sum(Qty * Price)/sum(Qty)
             –   FROM Trades_s KEEP 30 MINUTES
             –   GROUP BY Symbol;
        •    Join
             – CREATE SCHEMA Value_t
             –   INHERITS FROM Book_t (CurVal FLOAT, AvgVal FLOAT);
             – CREATE WINDOW BookValue_w
             –   SCHEMA Value_t KEEP LAST PER Symbol;
             – INSERT INTO BookValue_w
             –   SELECT B.Symbol, B.SharesHeld, B.BuyPrice,
             –          B.SharesHeld * L.Price, B.SharesHeld*V.Vwap
             –   FROM Book_w B, LastTrade_w L, Vwap_s V
             –   WHERE B.Symbol = L.Symbol and B.Symbol = V.Symbol;




34
epts
     event processing technical society




CCL Pattern Matching

        • MATCHING clause
             – INSERT INTO OrdersMatch_s
             –   SELECT B.ID, S.ID, B.Price
             –   FROM BuyOrders_s B, Trades_s T, SellOrders_s S,
             –   MATCHING [30 SECONDS: B, !T, S]
             –   ON B.Price = S.Price = T.Price;

        • Operators in patterns (cf. finite-state automata):
          • Three boolean operators: ! (not), & (and), | (or)
          • One temporal operator: , (followed by)
          • Temporal scope can be nested
             –          MATCHING [30 SECONDS: [10 SECONDS: B, !T], S]


35
epts
     event processing technical society




Modularity
        • Note: concrete syntax in future version of CCL
        • Group of streams can be abstracted into a module
             – CREATE MODULE OptionPrice
             –   CREATE INPUT STREAM Option_s;
             –   CREATE OUTPUT STREAM OptionPrice_s;
             –   CREATE PARAMETER FLOAT InterestRate = 0.05;
             –   // internal stream definitions
             – END MODULE;

        • Modules can be instantiated
             ● LOAD MODULE OptionPrice as LondonOptionPrices
             ●   STREAMS
             ●     Input = LondonOptions_s,
             ●     Output = LondonOptionPrices_s
             ●   PARAMETERS
             ●     InterestRate = 0.03;
36
epts
     event processing technical society




Other Features from Other Languages

        •    Esper (Java)/Nesper (C#)
             • Extensions of SQL
             • Embedded within host language
        •    AleriML
             • SPLASH: a C-like embedded language for writing nodes
             • Data structures: vectors, dictionaries, event caches
             • Mechanisms for discarding stored records (time, # records)
        •    StreamBase
             • Primitive types include nested records and vectors (lists)
             • LOCK/UNLOCK for holding and releasing records


37
epts
     event processing technical society




TUTORIAL OUTLINE

        Introduction                              Opher Etzion
        stream processing languages               Jon Riecke
             Rule oriented languages               Adrian Paschke   Languages
                                                                    Styles
        •    Break
        •    Agent oriented languages              Opher Etzion
        •    Temporal semantics                    Jon Riecke
                                                                    Advanced
        •    Integrated development environments   Jon Riecke
                                                                    Topics
        •    Formal approaches                     Adrian Paschke
        •    Conclusion                            Opher Etzion




38
epts
     event processing technical society




Production Rules




39
epts
     event processing technical society




Production Rules and CEP
     •   Production rules react to states changes (not events)
     •   PR systems with object model and external fact updates might
         be extended to CEP
         – Event types and classes are defined in the rule declarations
         – New event data instances are added to the fact base / working
           memory
             • might be an external fact base, e.g. a event queue
         – The instances of declarations are filtered and joined in the
           conditions via typed pattern matching and if they pass the condition
           list the action part of the rule is triggered
     •   Might be further extended with mechanisms like query
         languages, state models and temporal constraints
     •   Examples: TIBCO Business Events, Drools
40
epts
     event processing technical society




Production Rules

           •    A production rules is a statement of the form:

                                          “if Condition then Action”

           •    Members, e.g. OPS5, Clips, Jess, Drools, Fair Isaac Blaze
                Advisor, ILog jRules, CA Aion, Haley, ESI Logist, …


           •    Operational semantics: forward-chaining
                     – primitive update actions (assert, retract, …);




41
epts
     event processing technical society




Operational Semantics

        •         Operational processing
             –        forward-chaining order-independent execution (e.g. based
                      on variations of the RETE algorithm)
             –        Procedural order-dependent sequential execution in a
                      (compiled) execution environment (rule engine).


        •         Conflict resolution
             –        Refraction
             –        Priority
             –        Recency
             –        Specificity
             –        …

42
epts
     event processing technical society




     The Rete Pattern-Matching Algorithm

     •   Rete Pattern-Matching Algorithm (and variations of it)
         – matches facts against the patterns in rules to determine which rule
           conditions are satisfied
         – incremental evaluation of production rules' conditions



     •   Developed by Charles L. Forgy in the late 70s for OPS5 (Official
         Production System) shell
         – stores information about the antecedents in a network
         – in every cycle, it only checks for changes in the networks
          this greatly improves efficiency




43
epts
     event processing technical society




 OMG Production Rules Representation (OMG PRR)

     • Formal model for vendor-neutral rule-model
          representation in UML for production rules
     • Consortium of developers and supporters
        – Rule vendors (including Fair Isaac, IBM/ILOG, LibRT, Pega,
          Corticon, TIBCO)
        – Academic community (RuleML Group)
        – Related vendor community (Fujitsu)
     • PRR is currently “adopted" as a standard and in
        “finalization", meaning that it is in Beta, with a final
        version 1.0 in 2009.



44
epts
     event processing technical society




OMG Production Rules Representation

        • OMG MDA PIM model
        • PRR beta spec applies to a PRR Core using OMG
             MOF defined in UML
             – Extends UML so production rules are 1st class citizens
               alongside objects
             – Spec. excludes an explicit expression language
        • Can use other standards as expression language
             – e.g. W3C Rule Interchange Format (W3C RIF) and the
              production rule language of RuleML



45
epts
     event processing technical society




 OMG PRR ProductionRule Classes




46                                        if [condition] then [action]
epts
      event processing technical society




W3C Rule Interchange Format
• W3C Rule Interchange Format (W3C RIF)
     http://www.w3.org/2005/rules/wiki/RIF_Working_Group
• Goal
      Standardization of a web-based Rule Interchange Format (RIF)

• RIF 1.0 three dialects:
     – RIF Production Rules Dialect (RIF-PRD)
        • specifies the RIF production rules dialect to enable the interchange
          of production rules
     – RIF Basic Logic Dialect (RIF-BLD)
        • specifies a basic interchange format that allows logic rules (definite
          Horn rules with equality) to be exchanged
     – RIF Core
        • RIF-Core, a common subset of RIF-BLD and RIF-PRD
47
epts
     event processing technical society




 W3C RIF Production Rules Dialect
• Production Rules (Condition-Action)
     – Based on RIF condition language (from RIF Core)
     – Actions: Retract, Assert, Modify, Execute
     – Negation (inflationary negation)
• Syntax
     – Normative XML syntax
       • Might be concrete representation language for OMG PRR
     – Non-normative EBNF-based presentation syntax
       • for presentation purposes



48
epts
     event processing technical society




RIF Production Rule

     <Implies>
                     <if> FORMULA </if>
                     <then rif:ordered="yes">
                                 <Do>
                                          ACTION*
                                 </Do>
                     </then>                        Syntax close to PR
                                                     RuleML syntax
     </Implies>

49
epts
     event processing technical society




                   Event Condition Action Rules




50
epts
     event processing technical society




Event Condition Action Rules

        •    ECA Rule
                   “on Event if Condition do Action”;
             – Explicit complex event part; separated from conditions and
               actions
               • e.g. expresses customer order (event); check if credit card is
                 valid (condition)


        •    Evolved from active databases
             – extend databases with reactions, e.g. HiPac, ACCOOD, Chimera,
               ADL, COMPOSE, NAOS
             – Composite event algebra, e.g. SAMOS, COMPOSE, Snoop
               • Sequence | Disjunction | Xor | Conjunction |
                 Concurrent | Not | Any | Aperiodic | Periodic

51
epts
     event processing technical society




Time-point vs. Interval-based Semantics

             – Time-point semantics
                  • Event definition: B; (A;C) (Sequence)
                    event instance sequence EIS = b, a, c => detect event
                    event instance sequence EIS = a, b, c => detect event
             – Interval-based semantics
                  • Event definition B;(A;C) (Sequence)
                    event instance sequence EIS = b, a, c => detect event
                    event instance sequence EIS = a, b, c => do not detect event


                                T1            T2       T3       T4

                                          B        A        C

52
epts
     event processing technical society




Operational Semantics

        1.        Specification Phase
             –        Definition of event/action pattern e.g. by event algebra
             –        Based on declarative formalization or procedural implementation
             –        Defined over an atomic instant or an interval of time,
                      events/actions, situation, transition etc.
        2.        Selection
             –        Defines selection function to select one event from several
                      occurred events of a particular type in an event instance
                      sequence, e.g. “first”, “last”
             –        Crucial for the outcome of a reaction rule, since the events may
                      contain different (context) information, e.g. different message
                      payloads or sensing information




53
epts
      event processing technical society




Operational Semantics
     1.     Consumption
       –        Optionally defines which events are consumed after the detection of a
                composite event
       –        An event may contribute to the detection of several composite events, if
                it is not consumed
       –        Distinction in event messaging between “multiple receive” and “single
                receive”
       –        Events which can no longer contribute, e.g. are outdated, should be
                removed
     2.     Execution
       –        Actions might have an internal effect i.e. change the knowledge state
                leading to state transition from (pre)-condition state to post-condition
                state.
       –        Actions might have an external side effect


54
epts
     event processing technical society




Example: REWERSE MARS

        •    MARS (Modular Active Rules for the Semantic Web)
             – rule uses specialized heterogeneous event language (E), one
               or more query languages (Q), a test language (T), and an
               action language (A)
             – each of these languages and their constructs are described by
               metadata and an ontology of its semantics
             – well-defined interface for communication between the E, Q&T,
               and A components by variables
             – SNOOP (E), OWLQ (Q&T) and CCS - Calculus of
               Communicating Systems (A)




55
epts
     event processing technical society




Example: XChangeEQ

     •   ECA rules ON Event query IF Web query DO Action
         – the condition part is a declarative query to XML data
         – the event part is as well a declarative query to XML event
           data
              • distributed and web-based: events issued by web nodes (with both
                emission and reception times)
         – - action: simple or complex updates send to web nodes
     •   Complex events are expressed as (compound) queries to the
         event stream
         – temporal restrictions and event compositions
              • conjunction, disjunction, exclusion, quantifications, repetitions, and
                ranks
              • absolute temporal restrictions, also relative temporal restrictions
                (within)
56
epts
     event processing technical society




XChangeEQ
 •   The four dimensions of event querying are kept separated:
     – data extraction (eg oder number, book title)
     – event composition (eg oder of a dictionary and a grammar both from the same
       publisher)
     – temporal (and other) relationships (eg two orders within a week)
     – event accumulation (eg no default of paiement between the order of the two
       abovementioned buys)

 •   Equivalent Semantics:
     – Declarative semantics: Model and fixpoint theories
     – Operational semantics: CERA (Complex Event Relational Algebra)


 •   XChange^EQ operational semantics:
     – by language design, the system never wait forever for a complex event to possibly
       resume.
     – Incremental evaluation (refining RETE).
     – Built-in garbage collection of irrelevant complex events (eg events that cannot
       resume).

57
epts
     event processing technical society




Example: XChange Composite Event

        andthen [
             xchange:event {{
                            xchange:sender {"http://airline.com"},
                            cancellation-notification {{
                                          flight {{ number { var Number } }} }}
             }},
             xchange:event {{
                            xchange:sender {"http://airline.com"},
                            important {"Accomodation is not granted!"}
             }}
        ] within 2 h

58
epts
     event processing technical society




ruleCore Markup Language (rCML)

        •    XML language for ECA rules
        •    Complex Event defined in four sub-elements:
             – <detect-event>. The <detect-event> element specifies the
               event that is generated when the composite event is detected.
             – <no-detect-event>. The <no-detect-event> element specifies
               the event that is generated when the composite event can not
               be detected.
             – <event-selector> The <event-selector> element specifies a
               logical condition (or filter) for the composite event.
             – <detector>. The composite event detector consists of a number
               of event algebra operators (the SNOOP algebra operators) that
               describe the composite event.




59
epts
         event processing technical society




IBM AMIT Situation Manager Rule Language
     •   Definition of situations and their detection conditions
         – syntactically equivalent to (complex) event patterns
         – lifespans which hold references to events with relevant context for a
           particular pattern calculation
     •   Conceptual event model defines an event type generalization
         hierarchy
         – Root event type defines a set of standard attributes e.g.,
             • the event source, creation type, temporal dimensions like occurrence time,
               detection time and /or transaction time of an event, and quantifications such
               as status, priority, severity, count
         – Application specific event types inherit from root event type and define
           further user-defined attributes
         – Two types of event relationships
             • Based-on - a pure syntactical relationship to ease the development
               process. If an event A is 'basedon' event B, event A inherits the entire
               structure of event B and can extend it.
             • Reference - a relationship that allows deriving events without pattern
               detection. Useful for transformation and enrichment purposes.
60
epts
     event processing technical society




Example AMIT SMRL
 AMIT SMRL markup language provides an event algebra with semantic notations such as
 context (lifespan intervals), and semantic association (event group) to define situations

<eventTypes>
  <eventType name="doctorEnterRoom">
        <attributeType name="doctor" xsi:type="string"/>
        <attributeType name="room" xsi:type="integer"/> </eventType>
  <eventType name="patientEnterRoom">
        <attributeType name="patient" xsi:type="string"/>
        <attributeType name="r_o_o_m" xsi:type="integer"/>
  </eventType>
  <eventType name="lockRoom">
        <attributeType name="room" xsi:type="integer"/> </eventType>
  <eventType name="unlockRoom">
        <attributeType name="room" xsi:type="integer"/>
  </eventType>
</eventTypes>
<lifespan name="roomOpen">
        <initiator> <eventInitiator name="unlockRoom"/> </initiator>
        <terminator> <eventTerminator name="lockRoom"/> </terminator>
        <keyBy name="sameRoom"/>
</lifespan>
61
epts
     event processing technical society




(Reaction) RuleML

     •   General (reaction) rule form that can be specialized as needed
     •   Platform-independent XML-based rule interchange format
         – translation into platform-specific executable rule languages, e.g. Prova
     •   Three general execution styles:

         – Active: 'actively' polls/detects occurred events in global ECA style, e.g. by a ping on
           a service/system or a query on an internal or external event database
         – Messaging: Waits for incoming complex event message
         – Reasoning: KR event/action logic reasoning and transitions (as e.g. in Event
           Calculus, Situation Calculus, TAL formalizations)
     •   Appearance

         – Global: ‘globally’ defined reaction rule
         – Local: ‘locally’ defined reaction rule in a specific context



62
epts
     event processing technical society




General Syntax for Reaction Rules
               <Rule style="active" eval="strong">
                     <on>
                              <!-- event -->
                      </on>
                                 <if>
                                           <!-- condition -->
                                 </if>
                                 <do>
                                          <!--   action -->
                                 </do>
                              <ifPost>
                                       <!-- postcondition -->
                              </ifPost>
                    <doAlternative>
                            <!-- alternative/else action -->
                    </doAlternative>
               </Rule>
63
epts
     event processing technical society




Selected Reaction RuleML Extended Features
      •    Action Algebra:
             Succession (Ordered Succession of Actions), Choice
           (Non-Determenistic Choice), Flow (Parallel Flow), Loop
           (Loops)
      •    Event Algebra:
             Sequence (Ordered), Disjunction (Or) , Xor (Mutal
           Exclusive), Conjunction (And), Concurrent , Not, Any,
           Aperiodic, Periodic
      •    Support for event / action messaging
      •    Support for different detection, selection and consumption
           policies
      •    Support for intervals (Time, Event)
      •    Support for situations (States, Fluents)
      •    Support for external event query languages
      •    ...
64
epts
     event processing technical society




                                          10 minutes break
TUTORIAL OUTLINE

        Introduction                              Opher Etzion
        stream processing languages               Jon Riecke
         rule oriented languages                  Adrian Paschke   Languages
                                                                    Styles
              Break
        •    Agent oriented languages              Opher Etzion
        •    Temporal semantics                    Jon Riecke
                                                                    Advanced
        •    Integrated development environments   Jon Riecke
                                                                    Topics
        •    Formal approaches                     Adrian Paschke
        •    Conclusion                            Opher Etzion




65
epts
     event processing technical society




TUTORIAL OUTLINE

        Introduction                           Opher Etzion
        stream processing languages            Jon Riecke
         rule oriented languages               Adrian Paschke   Languages
                                                                 Styles
         Break
        • agent oriented languages              Opher Etzion
        • Temporal semantics                    Jon Riecke
                                                                 Advanced
        • Integrated development environments   Jon Riecke
                                                                 Topics
        • Formal approaches                     Adrian Paschke
        • Conclusion                            Opher Etzion




66
epts
     event processing technical society




EPA and EPN

        •    In David Luckham’s “the
             power of events” the term
             EPN (Event Processing
             Network) was used as a
             collection of EPAs (Event
             Processing Agents).
        •     The EPN describes the                                                                            c1
                                                                                                                        p1

                                                                                                                    admittance
                                                                                                                                  Filter: Diabetes
                                                                                                                                                            p5
                                                                                                                                                                    c5

                                           EPN Example
             “programming in the large”,                                                                                         Pattern detection:
                                                                                                                                  Alert physician
                                                                                                                                                              p6

                                                                                                                                                                 p7
                                                                                                                                                                   p12




             while each individual agent                Blood pressure reading
                                                                                                 p8
                                                                                                                 Enrich:
                                                                                                                                      p10
                                                                                                                                                  p11




             describes the
                                                                                          c2                                                  c6
                                                              p2                p9                             Is diabetic?
                                                                                                         p12
                                                                   Aggregator: Max.


             “programming in the
                                                                                                                                      c7
                                                  Fever C                                      p14          Translate:                      p13
                                                                                     c3                     C2F fever                                         p19
                                                Fever F             p3


             small”.                                               p4
                                                                                               Fever F

                                                                                                       c8
                                                                                                             p15
                                                                                                                                 Pattern Detection:
                                                                                                                                     Alert Nurse

                                                                                               p16
                                                                                                                                                            p20
                                                                                                     Pattern Detection:                p18
                                                                                                     Continuous Fever


                                                                                                                       p17       c9




                                                 IBM Haifa Research Lab – Event Processing                                                          © 2008 IBM Corporation



67
epts
           event processing technical society




Some EPN, event flow and data flow oriented languages

                               Agent Logic                                                                                               Event Zero      Oracle/BEA




                   Spade (IBM
                          System S)
        IBM InfoSphere Streams                                                                                                              Streambase
 A SPADE program builds a data-flow network out of operators



        source     stream   processing                                                                      sink
       operator              operator                                                                     operator
                                                                                                   m
                                                      str




                                                                                                    ea
                                                         ea




                                                                                                 str
                                                        m




                                                              processing   stream   processing
                                                               operator              operator
                                                                                                   stre




                                                  m
                                               ea
                                                                                                     am




                                         str

                   source                                                                                   sink
                  operator                                                                                operator




68
 194    Tutorial on the SPADE language                                                                    © 2008, 2009 IBM Corporation
epts
     event processing technical society




What is the idea ?

        •    The EPN (event processing network) represents the event
             processing application as a directed graph:
             – The nodes represent event processing agents (and states in some
               models)
             – The edges represent either individual events or event streams
               (depends on the processing type). A generalized EPN may support
               both.
        •    The coverage of EPN varies
             – It may cover some language operator (e.g. SQL like language)
             – It may take a broader approach and cover also routing decisions, pub/
               sub etc…
             – EPN can have a specific programming model, or an hybrid
               programming model, where one or more languages represent the
               agents.




69
epts
     event processing technical society




Anatomy of Agents




                               Performs the processing
        Selects the events that are
                                               Derives output events
        relevant for the processing detect patterns)
                               (e.g.




                          Selector        Processor          Deriver


70
epts
     event processing technical society




Event Processing Specialized Agent types
                                                  Event Processing Agent




                                     Filter     Transform Detect Pattern   Route




             Translate                    Aggregate    Split    Compose     Enrich   Project




71
epts
      event processing technical society




Routing – channel EPA


     Itinerary-based routing




     Subscription-based
     routing




     Intelligent routing




72
epts
     event processing technical society



Making the agents context sensitive
                                          Time interval that starts by: {event, clock, offset from event}
                                          and terminates by: {event, clock, offset from event, count of
                                          certain events} – each can have multiple instances (e.g. several
                                          events, periodic clock, multiple offsets}
                             When?
                                          Examples: [9:00AM, angry-customer-alert], [angry-
                                          customer-alert, +1 hour], [request – 2 hours,
                                          request]

                                                   Spatial dimensions/ abstraction
                                Where?         Inside building A, In Italy, within 1KM from here
                                               (given GPS coordinate)

                                                     An entity or a collection of entities that determine the partition
                                                     of the context
                                   Who?          Class of customer, customer + order, large order…



                                                     The event happens when some state is in effect

                            Which state?
                                                     Traffic Jam, Red Alert, Bullish Market




73
epts
     event processing technical society




Filtering

         Many of the raw events are not
         relevant for processing, in some
         applications only a sample is
         selected for processing.




74
epts
     event processing technical society




Transformation

             Translation / projection:
             Translation and simple
             derivations


            Aggregation/ composition:
            statistical aggregator or
            concatenating events, can be
            stand alone agent or a scalar
            derivation. May be in network
            edge.

              Splitting: Splits events to
              multiple events

75
epts
     event processing technical society




Enrichment


           Enriches the content of events
           from reference data in databases,
           spreadsheets, Email messages,
           text files etc..




76
epts
     event processing technical society




Pattern Matching                                        Pattern definition
                                                        example




                      In many cases, the agent does not
                      consume raw events, but event that
                      are derived from pattern matching.


                            DEBS 2008 tutorial on
                            event processing patterns
                            provided a deep dive
77
epts
     event processing technical society




Dimensions - I

        • Meta-data
             – EPN and agent definitions may be defined as meta-data
             – Event schema is implementation dependent
        •      State
             – State can be represented as another node in the EPN
             – Some EPAs may be state oriented, e.g. the input event is
               change in state
             – State is one of the context dimensions



78
epts
     event processing technical society




Dimensions - II

        • Execution model:
             – EPN representation is natural one for
               concurrency/parallelism since the dependencies are
               explicit
             – Since it models explicitly the end-to-end event flow, it
              serve as a convenient model for QoS related
              optimizations (latency, throughput, real-time constraints)
        • Programming model:
             – Can support multiple programming styles
             – Can support all types of event processing functionality



79
epts
     event processing technical society
                                                                                                 p1                              p5
                                                                                        c1                                              c5
                                                                                                           Filter: Diabetes
                                                                                             admittance
                                                                                                                                 p6
EPN example                                                                                               Pattern detection:            p12
                                                                                                           Alert physician            p7


                                  Blood pressure reading                                                                   p11
                                                                          p8                                   p10
                                                                   c2                     Enrich:
                                          p2                                            Is diabetic?                   c6
                                                         p9
                                                                                  p12
                                               Aggregator: Max.
                                                                                                               c7
                           Fever C                                      p14          Translate:                      p13
                                                              c3                     C2F fever                                   p19
                       Fever F                 p3
                                                                        Fever F       p15
                                                                                                          Pattern Detection:
                                               p4                                                             Alert Nurse
                                                                                c8
                                                                        p16
                                                                                                                                 p20
                                                                              Pattern Detection:                p18
                                                                              Continuous Fever


                                                                                                p17       c9




80
epts
     event processing technical society




Event Zero example




81
epts
     event processing technical society




System S Example


                                      Congestive Heart Failure
                                                              Freq                                             alert
                                                                                            Arrhythmia
                                                             Analyzer

                                                     QRS          RR              P and T


                                     Source PE               Glucose
                                                                                                                  alert
                                                             BP S/D                                      CHF


                                                                   Weight trend                                            IHE
                                                                    detector
                                                                                                                          Adapter




                                                                                                                               *Daby Sow
82
epts
     event processing technical society




Agent Logic example




83
epts
     event processing technical society




TUTORIAL OUTLINE

        Introduction                           Opher Etzion
        stream processing languages            Jon Riecke
         rule oriented languages               Adrian Paschke   Languages
                                                                 Styles
         Break
        agent oriented languages               Opher Etzion
        • Temporal semantics                    Jon Riecke
                                                                 Advanced
        • Integrated development environments   Jon Riecke
                                                                 Topics
        • Formal approaches                     Adrian Paschke
        • Conclusion                            Opher Etzion




84
epts
     event processing technical society




Varieties of Temporal Semantics

        •    Do events carry timestamps? Where does the timestamp
             come from (CEP system or external world)?
        •    Granularity? Second, millisecond, microsecond, …?
        •    Point in time or interval of time?
        •    Can events arrive out-of-order from a single data source?
        •    Can events be reordered if they arrive out-of-order?
             • How long does the system wait?
             • What happens if system moves ahead, and old event arrives?
        •    In distributed case, is there a difference between emission
             and reception time?




85
epts
     event processing technical society




Simple Temporal Semantics

        • Events arrive in order from a single data source
        • Events have timestamps, but not reordered
        • Advantage
             • Process as soon as possible
        • Disadvantages
             • Non-deterministic
             • Hard to manage if time really matters
             • Emission times de-synchronized in distributed case


86
epts
     event processing technical society




Global Clock

        •    Events have timestamps consistent across data sources
        •    Wait for events for specified time; discard if wait too long
        •    Advantages
             •    Deterministic
             •    Time might be relevant
        •    Disadvantages
             •    Speed of processing
             •    Buffering required for reordering events




87
epts
     event processing technical society




TUTORIAL OUTLINE

        Introduction                           Opher Etzion
        stream processing languages            Jon Riecke
         rule oriented languages               Adrian Paschke   Languages
                                                                 Styles
         Break
        agent oriented languages               Opher Etzion
         Temporal semantics                    Jon Riecke
                                                                 Advanced
        • Integrated development environments   Jon Riecke
                                                                 Topics
        • Formal approaches                     Adrian Paschke
        • Conclusion                            Opher Etzion




88
epts
     event processing technical society




Integrated Development Environments (IDEs)

        • Two main forms
             • Text based
             • Visual or graphical based
        • Not an either-or; most systems have some
             combination of text and visual display of dataflow




89
epts
     event processing technical society




Visual IDEs

        • Streams = nodes, flows = edges
        • Drag-and-drop streams from palette onto canvas
             and wire together
        • Configure streams with property sheets or text
        • Advantages: simple metaphor; easy to see flow
        • Disadvantages: managing screen real-estate




90
epts
     event processing technical society




Visual IDE Example




91
epts
     event processing technical society




Text-based IDEs

        • Streams represented in text
        • Advantages
             • Appeals to programmers
             • Can be quicker to write code
        • Disadvantages
             • Hard to see the data flow
             • Hard to find bottlenecks in program



92
epts
     event processing technical society




Text-based IDE




93
epts
     event processing technical society




Another Text-based IDE




94
epts
     event processing technical society




Advanced Features in IDEs

        • Ability to start/stop programs
        • Debugging, including viewing intermediate events
             and setting breakpoints
        • Tests, including ability to control data feeds
        • Dashboards for displaying results




95
epts
     event processing technical society




Breakpoints and Debugging




96
epts
     event processing technical society




Performance Monitoring




97
epts
     event processing technical society




Dashboard Construction




98
epts
     event processing technical society




Dashboard Construction




99
epts
      event processing technical society




Assessment: IDEs still have a ways to go

         • Need more features that programmers have come
              to expect
              • Single-stepping with line numbers
              • Examination of events and data
              • Forward and backward replay of time
         • Need to be integrated into more dashboards and
              other visualization tools




100
epts
      event processing technical society




TUTORIAL OUTLINE

         Introduction                           Opher Etzion
         stream processing languages            Jon Riecke
          rule oriented languages               Adrian Paschke   Languages
                                                                  Styles
          Break
         agent oriented languages               Opher Etzion
          Temporal semantics                    Jon Riecke
                                                                  Advanced
          Integrated development environments   Jon Riecke
                                                                  Topics
         • Formal approaches                     Adrian Paschke
         • Conclusion                            Opher Etzion




101
epts
      event processing technical society




                                    Formal Approaches
                               Temporal Event/Action Logics for
                                   Reasoning on Changes




102
epts
      event processing technical society




(Temporal) Event Logics

  • (Temporal) event/action logics
      – Events with effects on changeable properties / states / fluents
      – Focus: reasoning on effects of events/actions on knowledge
        states and properties
      – Not Complex Event Processing Languages, but can be used
        in reasoning on/with complex events
  • Members e.g.
      – event calculus and variants,
      – situation calculus,
      – features and fluents calculus,
      – various (temporal) action languages (TAL),
      – fluent calculi and versatile event logics.
103
epts
      event processing technical society




The Situation Calculus

 •    A situation is a snapshot of the
      world at some point in time
 •    Every true or false statement is
      made with respect to a
      particular situation




       • When an agent performs an action A is situation S1, the
       result is a new situation S2
       • McCarthy 1963, McCarthy & Hayes 1969, Green 1969


104
epts
      event processing technical society




Event Calculus
  •   Kowalski and Sergot’s EC
      – a formalism for temporal reasoning about events and their effects
           • computation of earlier events (long-term "historical" perspective)
      – model of change
           • events happen at time-points and initiate and/or terminate properties (time-
             varying fluents) of the world
      – Law of inertia:
        Things normally tend to stay the same

  •   Main differences in EC to SC
      – branching time in SC vs. linear time in EC
      – explicit notion of previous state of the world / situation in SC
      – state transitions are functions in SC



105
epts
      event processing technical society




Classical Event Calculus Example
  •   EC Basic Axioms:
      – happens(E,T)                                 event E happens at time point T
      – initiates(E,F,T)                             event E initiates fluent F for all time>T
      – terminates(E,F,T)                            event E terminates fluent F for all time>T
      – holdsAt(F,T)                       fluent F holds at time point T


  •   Many EC Extensions, e.g.:
      – valueAt(P,T,X)       parameter P has changeable value X at time point T
      – planned(E,T)         event E is believed to happen at time point T

  Example:
      initiates(stopService,serviceUnavailable,T)                         stopService startService
      terminates(startService,serviceUnavailable,T)
      happens(stopService,t1); happens(startService,t5)
                                                                               t1 t3 ? t5     t7 ?
      holdsAt(serviceUnavailable,t3)?                       true
      holdsAt(serviceUnavailable,t7)?                       false

106
epts
       event processing technical society




Example: Interval-based Event Calculus Event Algebra

        – Event initiate and terminate Situations (Fluents) which hold at an time interval
        – Interval-based Event Calculus semantics (model-theory + proof theory)
          based on time intervals modeled as fluents
                      I: Ti x Fl  {true, false}
        – Example: B;(A;C) (Sequence)


                                      T1        T2       T3                             T4

                                            A        B                 C
      (A;B;C) [T1,T3]) <=
                       holdsInterval([a,b] [T1,T2]) Λ holdsInterval([b,c] [T2,T3]) Λ [T1,T2]<=[T2,T3]

        – Rule-based implementation of EC event algebra, e.g. as meta logic program
        – Rule-based arithmetic involving times and durations, e.g. Allen’s interval logic
107
epts
      event processing technical society




 Combinations of Rule Types
1. Rules that influence the operational / decision processes:
   • Derivation rules (deduction rules): establish / derive new
       information that is used e.g. in a decision process.
   • Reaction rules that establish when certain activities should place
       (e.g. ECA rules)

2. Constraints on system/organisation's structure, behavior or information:
   • Structural constraints (e.g. deontic assignments), State constraints,
      Process / flow constraints




108
epts
      event processing technical society




Example: Prova Messaging Reaction Rules
 %Event Message
 rcvMsg(EventID, jms, Requester, acl_query-ref, Task) :-
   %Condition(s) – “find available service”
   available(Service),      variable data binding
   %Action Message – “load task on service”
   sendMsg(Sub-ID,jms,Service,acl_query-ref, load(Task)).

 % Derivation rules
 available(S) :-
   service(S),
   not(maintenance(S)),
   not(loaded(S),T).

 maintenance(S):- …                        automatic rule chaining

 loaded(S) :- …
109
epts
      event processing technical society
                                           Messaging Reaction Rules in Prova

 •    Send a message
 sendMsg(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)

 •    Receive a message
 rcvMsg(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)

 •    Receive multiple messages
 rcvMult(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)


 Syntax:
  – XID is the conversation / event instance sequence identifier
  – Protocol: ESB transport protocols (>30 e.g. self, jade, jms, soap,…)
  – Agent: denotes the target or sender of the message
  – Performative: pragmatic context, e.g. FIPA ACL primitives
  – [Predicate|Args] or Predicate(Arg1,..,Argn): Message payload


110
epts
      event processing technical society




TUTORIAL OUTLINE

         Introduction                           Opher Etzion
         stream processing languages            Jon Riecke
          rule oriented languages               Adrian Paschke   Languages
                                                                  Styles
          Break
         agent oriented languages               Opher Etzion
          Temporal semantics                    Jon Riecke
                                                                  Advanced
          Integrated development environments   Jon Riecke
                                                                  Topics
          Formal approaches                     Adrian Paschke
               Conclusion                        Opher Etzion




111
epts
      event processing technical society




Event Processing Languages – insights from this tutorial


                                                     epts
                                                     event processing technical society

                epts
                event processing technical society


        An Observation
                    Dimensions toTower symbolizeslanguages
                         The Babylon describe the tendency
                                                                   Of humanity to talk in multiple languages.


                                      • Events Data and Meta-data
                                      • State
                                      • no different: most languages in the model
             The Event Processing area is Computation/execution industry really follow
             the hammer and nails syndrome – and extended existing approaches

                                      • Programming model
             • imperative script language
             • SQL extensions
             • Extension of inference rule language



                                   It does not seem that we’ll succeed to settle
                                   In the near future around a single programming style



                                         The epts language analysis workgroup is aimed to understand the various styles
                                         And extract common functions that can be used to define what is an event
                                         processing language; this tutorial is an interim report
112
epts
          event processing technical society




Towards event processing languages standard ?
          • Eventually – a single language ?
                      1st generation
                                                    SQL standard approved
          of relational DBMS products

 Mid 1970-ies                                  1990

                      1st generation
                                               EP standard approved
                    of EP products

      Mid 200-ies                               ?



      •      OR – variety of language style standards ?


      •        Meta-language as an intermediate standard ?
113
epts
      event processing technical society




The epts “Event Processing Language Analysis” workgroup
  Mission:
  The purpose of this working group is to conduct a study of the features that exist in the contemporary and planned languages
  in the area of event processing (including event stream processing).
  The goal is to understand the functional properties that exist in event processing languages and abstract out semantic functions,
  regardless of syntax and implementation considerations.
  This is a first step in a way to determine future possible standardization in the event processing area

       WG Leaders: Opher Etzion (IBM) and Jon Riecke (Aleri)
       Members:
       •    Alex Kozlenkov (Betfair)
       •    Adrian Paschke (FU Berlin)
       •    Arno Jacobsen (U of Toronto)
       •    Bala Maniymaran (U of Toronto)
       •    Bob Hagmann (Aleri)
       •    David Tucker (EventZero)
       •    Dermot McPeake (FirstDerivatives)
       •    Francois Bry (LMU)
       •    Guy Sharon (IBM)
       •    Louis Lovas (Progress/Apama)
       •    Michael Eckert (TIBCO)
       •    Pedro Bizarro (University of Coimbra)
       •    Richard Tibbetts (Streambase)
       •    Robert McKeown (IBM)
       •    Susan Urban (Texas Tech. University)
       •    Simon Courtenage (Westminister U.)
       •    Serge Mankovskii (CA)

114
epts
                                     event processing technical society




If you want to participate in these discussions, contribute to this
topic and/or others topics, and be part of the community effort to
advance the state of the art and state of the practice:
JOIN EPTS
For details: http://www.ep-ts.com/

More Related Content

Viewers also liked

Chapter 12
Chapter 12Chapter 12
Chapter 12
cclay3
 
Module 3 Scanning
Module 3   ScanningModule 3   Scanning
Module 3 Scanning
leminhvuong
 

Viewers also liked (20)

RuleML2011 CEP Standards Reference Model
RuleML2011 CEP Standards Reference ModelRuleML2011 CEP Standards Reference Model
RuleML2011 CEP Standards Reference Model
 
NS2--Event Scheduler
NS2--Event SchedulerNS2--Event Scheduler
NS2--Event Scheduler
 
Comparative Analysis of Personal Firewalls
Comparative Analysis of Personal FirewallsComparative Analysis of Personal Firewalls
Comparative Analysis of Personal Firewalls
 
Session hijacking
Session hijackingSession hijacking
Session hijacking
 
Access control attacks by nor liyana binti azman
Access control attacks by nor liyana binti azmanAccess control attacks by nor liyana binti azman
Access control attacks by nor liyana binti azman
 
Installing Complex Event Processing On Linux
Installing Complex Event Processing On LinuxInstalling Complex Event Processing On Linux
Installing Complex Event Processing On Linux
 
Reactconf 2014 - Event Stream Processing
Reactconf 2014 - Event Stream ProcessingReactconf 2014 - Event Stream Processing
Reactconf 2014 - Event Stream Processing
 
Tutorial in DEBS 2008 - Event Processing Patterns
Tutorial in DEBS 2008 - Event Processing PatternsTutorial in DEBS 2008 - Event Processing Patterns
Tutorial in DEBS 2008 - Event Processing Patterns
 
Debs 2011 tutorial on non functional properties of event processing
Debs 2011 tutorial  on non functional properties of event processingDebs 2011 tutorial  on non functional properties of event processing
Debs 2011 tutorial on non functional properties of event processing
 
CyberLab CCEH Session - 3 Scanning Networks
CyberLab CCEH Session - 3 Scanning NetworksCyberLab CCEH Session - 3 Scanning Networks
CyberLab CCEH Session - 3 Scanning Networks
 
Ceh v8 labs module 03 scanning networks
Ceh v8 labs module 03 scanning networksCeh v8 labs module 03 scanning networks
Ceh v8 labs module 03 scanning networks
 
Complex Event Processing with Esper and WSO2 ESB
Complex Event Processing with Esper and WSO2 ESBComplex Event Processing with Esper and WSO2 ESB
Complex Event Processing with Esper and WSO2 ESB
 
Chapter 12
Chapter 12Chapter 12
Chapter 12
 
Nmap scripting engine
Nmap scripting engineNmap scripting engine
Nmap scripting engine
 
Why Data Virtualization Is Good For Big Data Analytics?
Why Data Virtualization Is Good For Big Data Analytics?Why Data Virtualization Is Good For Big Data Analytics?
Why Data Virtualization Is Good For Big Data Analytics?
 
Tutoriel esper
Tutoriel esperTutoriel esper
Tutoriel esper
 
Analizadores de Protocolos
Analizadores de ProtocolosAnalizadores de Protocolos
Analizadores de Protocolos
 
Scanning with nmap
Scanning with nmapScanning with nmap
Scanning with nmap
 
Module 3 Scanning
Module 3   ScanningModule 3   Scanning
Module 3 Scanning
 
Port Scanning Overview
Port Scanning  OverviewPort Scanning  Overview
Port Scanning Overview
 

Similar to Debs2009 Event Processing Languages Tutorial

Essence: A Common Ground for Flexible Methods
Essence: A Common Ground for Flexible MethodsEssence: A Common Ground for Flexible Methods
Essence: A Common Ground for Flexible Methods
Ed Seidewitz
 
Genome in a Bottle Consortium Workshop Welcome Aug. 16
Genome in a Bottle Consortium Workshop Welcome Aug. 16Genome in a Bottle Consortium Workshop Welcome Aug. 16
Genome in a Bottle Consortium Workshop Welcome Aug. 16
GenomeInABottle
 
Parallel Programming in .NET
Parallel Programming in .NETParallel Programming in .NET
Parallel Programming in .NET
SANKARSAN BOSE
 
Differentiation sample english unit plan
Differentiation   sample english unit planDifferentiation   sample english unit plan
Differentiation sample english unit plan
luvreading
 
Pain points for preservation services / workflows in repositories
Pain points for preservation services /  workflows in repositories Pain points for preservation services /  workflows in repositories
Pain points for preservation services / workflows in repositories
prwheatley
 
ISO/IEc 15504/SPICE Status
ISO/IEc 15504/SPICE StatusISO/IEc 15504/SPICE Status
ISO/IEc 15504/SPICE Status
Alec Dorling
 
Building Serious Games for Medical Intervention and Training
Building Serious Games for Medical Intervention and TrainingBuilding Serious Games for Medical Intervention and Training
Building Serious Games for Medical Intervention and Training
Brock Dubbels
 

Similar to Debs2009 Event Processing Languages Tutorial (20)

Being a Language Technology Specialist
Being a Language Technology SpecialistBeing a Language Technology Specialist
Being a Language Technology Specialist
 
Seronto Process
Seronto ProcessSeronto Process
Seronto Process
 
Agile Business Analysis - The Key to Effective Requirements on Agile Projects
Agile Business Analysis - The Key to Effective Requirements on Agile ProjectsAgile Business Analysis - The Key to Effective Requirements on Agile Projects
Agile Business Analysis - The Key to Effective Requirements on Agile Projects
 
Agile led alfresco implementation jan 2011 (final)
Agile led alfresco implementation jan 2011 (final)Agile led alfresco implementation jan 2011 (final)
Agile led alfresco implementation jan 2011 (final)
 
Essence: A Common Ground for Flexible Methods
Essence: A Common Ground for Flexible MethodsEssence: A Common Ground for Flexible Methods
Essence: A Common Ground for Flexible Methods
 
Genome in a Bottle Consortium Workshop Welcome Aug. 16
Genome in a Bottle Consortium Workshop Welcome Aug. 16Genome in a Bottle Consortium Workshop Welcome Aug. 16
Genome in a Bottle Consortium Workshop Welcome Aug. 16
 
A middleware platform_to_federate_complex_event_processing
A middleware platform_to_federate_complex_event_processingA middleware platform_to_federate_complex_event_processing
A middleware platform_to_federate_complex_event_processing
 
The Straight Jacket of Agile Iteration
The Straight Jacket of Agile IterationThe Straight Jacket of Agile Iteration
The Straight Jacket of Agile Iteration
 
Arrows Group Event 06
Arrows Group Event 06Arrows Group Event 06
Arrows Group Event 06
 
Parallel Programming in .NET
Parallel Programming in .NETParallel Programming in .NET
Parallel Programming in .NET
 
Differentiation sample english unit plan
Differentiation   sample english unit planDifferentiation   sample english unit plan
Differentiation sample english unit plan
 
Welcome To Etech
Welcome To EtechWelcome To Etech
Welcome To Etech
 
Pain points for preservation services / workflows in repositories
Pain points for preservation services /  workflows in repositories Pain points for preservation services /  workflows in repositories
Pain points for preservation services / workflows in repositories
 
ISO/IEc 15504/SPICE Status
ISO/IEc 15504/SPICE StatusISO/IEc 15504/SPICE Status
ISO/IEc 15504/SPICE Status
 
Building Serious Games for Medical Intervention and Training
Building Serious Games for Medical Intervention and TrainingBuilding Serious Games for Medical Intervention and Training
Building Serious Games for Medical Intervention and Training
 
Complex Event Processing: What?, Why?, How?
Complex Event Processing: What?, Why?, How?Complex Event Processing: What?, Why?, How?
Complex Event Processing: What?, Why?, How?
 
Scientific data management from the lab to the web
Scientific data management   from the lab to the webScientific data management   from the lab to the web
Scientific data management from the lab to the web
 
A View on eScience
A View on eScienceA View on eScience
A View on eScience
 
Impact centre of_competence_for_workshop_ocr_rouen_march_2011[1]
Impact centre of_competence_for_workshop_ocr_rouen_march_2011[1]Impact centre of_competence_for_workshop_ocr_rouen_march_2011[1]
Impact centre of_competence_for_workshop_ocr_rouen_march_2011[1]
 
Debs Presentation 2009 July62009
Debs Presentation 2009 July62009Debs Presentation 2009 July62009
Debs Presentation 2009 July62009
 

More from Opher Etzion

On Internet of Everything and Personalization. Talk in INTEROP 2014
On Internet of Everything and Personalization. Talk in INTEROP 2014On Internet of Everything and Personalization. Talk in INTEROP 2014
On Internet of Everything and Personalization. Talk in INTEROP 2014
Opher Etzion
 
The Internet of Things and some introduction to the Technological Empowerment...
The Internet of Things and some introduction to the Technological Empowerment...The Internet of Things and some introduction to the Technological Empowerment...
The Internet of Things and some introduction to the Technological Empowerment...
Opher Etzion
 
Debs 2013 tutorial : Why is event-driven thinking different from traditional ...
Debs 2013 tutorial : Why is event-driven thinking different from traditional ...Debs 2013 tutorial : Why is event-driven thinking different from traditional ...
Debs 2013 tutorial : Why is event-driven thinking different from traditional ...
Opher Etzion
 

More from Opher Etzion (20)

DEBS 2019 tutorial : correctness and consistency of event-based systems
DEBS 2019 tutorial  : correctness and consistency of event-based systems DEBS 2019 tutorial  : correctness and consistency of event-based systems
DEBS 2019 tutorial : correctness and consistency of event-based systems
 
Sw architectures 2018 on microservices and eda
Sw architectures 2018    on microservices and edaSw architectures 2018    on microservices and eda
Sw architectures 2018 on microservices and eda
 
ER 2017 tutorial - On Paradoxes, Autonomous Systems and dilemmas
ER 2017 tutorial - On Paradoxes, Autonomous Systems and dilemmasER 2017 tutorial - On Paradoxes, Autonomous Systems and dilemmas
ER 2017 tutorial - On Paradoxes, Autonomous Systems and dilemmas
 
Event processing within the human body - Tutorial
Event processing within the human body - Tutorial Event processing within the human body - Tutorial
Event processing within the human body - Tutorial
 
DEBS 2015 tutorial When Artificial Intelligence meets the Internet of Things
DEBS 2015 tutorial   When Artificial Intelligence meets the Internet of ThingsDEBS 2015 tutorial   When Artificial Intelligence meets the Internet of Things
DEBS 2015 tutorial When Artificial Intelligence meets the Internet of Things
 
Dynamic stories
Dynamic storiesDynamic stories
Dynamic stories
 
Has Internet of Things really happened?
Has Internet of Things really happened? Has Internet of Things really happened?
Has Internet of Things really happened?
 
On the personalization of event-based systems
On the personalization of event-based systems On the personalization of event-based systems
On the personalization of event-based systems
 
On Internet of Everything and Personalization. Talk in INTEROP 2014
On Internet of Everything and Personalization. Talk in INTEROP 2014On Internet of Everything and Personalization. Talk in INTEROP 2014
On Internet of Everything and Personalization. Talk in INTEROP 2014
 
Introduction to the institute of technological empowerment
Introduction to the institute of technological empowermentIntroduction to the institute of technological empowerment
Introduction to the institute of technological empowerment
 
DEBS 2014 tutorial on the Internet of Everything.
DEBS 2014 tutorial  on the Internet of Everything. DEBS 2014 tutorial  on the Internet of Everything.
DEBS 2014 tutorial on the Internet of Everything.
 
The Internet of Things and some introduction to the Technological Empowerment...
The Internet of Things and some introduction to the Technological Empowerment...The Internet of Things and some introduction to the Technological Empowerment...
The Internet of Things and some introduction to the Technological Empowerment...
 
ER 2013 tutorial: modeling the event driven world
ER 2013 tutorial:  modeling the event driven world ER 2013 tutorial:  modeling the event driven world
ER 2013 tutorial: modeling the event driven world
 
Event semantics and model - multimedia events workshop
Event semantics and model -  multimedia events workshopEvent semantics and model -  multimedia events workshop
Event semantics and model - multimedia events workshop
 
Debs 2013 tutorial : Why is event-driven thinking different from traditional ...
Debs 2013 tutorial : Why is event-driven thinking different from traditional ...Debs 2013 tutorial : Why is event-driven thinking different from traditional ...
Debs 2013 tutorial : Why is event-driven thinking different from traditional ...
 
Debs 2012 gong show immortality
Debs 2012 gong show immortalityDebs 2012 gong show immortality
Debs 2012 gong show immortality
 
Debs 2012 basic proactive
Debs 2012 basic proactiveDebs 2012 basic proactive
Debs 2012 basic proactive
 
Debs 2012 uncertainty tutorial
Debs 2012 uncertainty tutorialDebs 2012 uncertainty tutorial
Debs 2012 uncertainty tutorial
 
Proactive eth talk
Proactive eth talkProactive eth talk
Proactive eth talk
 
Aaai 2011 event processing tutorial
Aaai 2011 event processing tutorialAaai 2011 event processing tutorial
Aaai 2011 event processing tutorial
 

Recently uploaded

Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
FIDO Alliance
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
FIDO Alliance
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
panagenda
 
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
Muhammad Subhan
 

Recently uploaded (20)

How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfHow we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdf
 
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdfLinux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
 
Design and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data ScienceDesign and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data Science
 
2024 May Patch Tuesday
2024 May Patch Tuesday2024 May Patch Tuesday
2024 May Patch Tuesday
 
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
 
Design Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxDesign Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptx
 
Top 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development CompaniesTop 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development Companies
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
 
TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024
 
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?
 
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdfThe Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdfWhere to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
 
Generative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfGenerative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdf
 
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
 
JavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate GuideJavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate Guide
 

Debs2009 Event Processing Languages Tutorial

  • 1. epts event processing technical society Event Processing Language Tutorial François Bry, Michael Eckert, Opher Etzion, Adrian Paschke, Jon Riecke on behalf of the epts languages analysis working group
  • 2. epts event processing technical society TUTORIAL OUTLINE • Introduction Opher Etzion • stream processing languages Jon Riecke • rule oriented languages Adrian Paschke Languages Styles Break • Agent oriented languages Opher Etzion • Temporal semantics Jon Riecke Advanced • Integrated development environments Jon Riecke Topics • Formal approaches Adrian Paschke • Conclusion Opher Etzion 2
  • 3. epts event processing technical society TUTORIAL OUTLINE • Introduction Opher Etzion • stream processing languages Jon Riecke • rule oriented languages Adrian Paschke Languages Styles Break • Agent oriented languages Opher Etzion • Temporal semantics Jon Riecke Advanced • Integrated development environments Jon Riecke Topics • Formal approaches Adrian Paschke • Conclusion Opher Etzion 3
  • 4. epts event processing technical society What is it all about? Filter, transform, enrich, route, Detect patterns, derive events Event Processing Event Event Producer Consumer Generate and emit Consume and react events perform operations on to either raw or events complex events Event processing has gained a lot of interest in recent years; according to analysts it is the fastest growing segment of middleware software In this tutorial we are Pattern exploring the language Match issue – how the various event processing functions are expressed? 4
  • 5. epts event processing technical society An Observation The Babylon Tower symbolizes the tendency Of humanity to talk in multiple languages. The Event Processing area is no different: most languages in the industry really follow the hammer and nails syndrome – and extended existing approaches • imperative script language • SQL extensions • Extension of inference rule language It does not seem that we’ll succeed to settle In the near future around a single programming style The epts language analysis workgroup is aimed to understand the various styles And extract common functions that can be used to define what is an event processing language; this tutorial is an interim report 5
  • 6. epts event processing technical society Existing Styles for EP languages (samples) Inference TIBCO Rules Prova WBE XChangeEQ ECA Rules RuleCore AMiT Starview Agent Agent Logic Oriented EventZero Spade SQL Aleri extension Streambase Coral8 State Oracle oriented Esper Imperative/ Netcool Impact Script Based Apama * - if we add simple and mediated event processing the picture is even more diversified 6
  • 7. epts event processing technical society Dimensions to describe languages • Events Data and Meta-data • State • Computation/execution model • Programming model 7
  • 8. epts event processing technical society The event Data and meta-data dimension • Event structures supported (structured, semi- structured, unstructured) • Data types supported for event’s payload • Event identifiers/identities • Event time-stamps and ordering • Event relations • Other meta-data entities in the language 8
  • 9. epts event processing technical society The state dimension • State of events / data – Events and other data in states – Life-cycle, duration policies – Explicit visibility and manipulation of states • State of execution – Context-related grouping (e.g. time windows) – Explicit visibility into state of execution 9
  • 10. epts event processing technical society The execution model dimension • Execution timing – immediate or deferred ? • Guarantees: real-time, determinism, latency ? • Time point or time interval semantics • Concurrency – implicit / explicit • Event at a time / set at a time processing • Explicit event flow 10
  • 11. epts event processing technical society The programming model dimension • Programming style • Abstractions • Pattern matching • Enrichment capabilities • Transformation capabilities • Filter capabilities 11
  • 12. epts event processing technical society TUTORIAL OUTLINE Introduction Opher Etzion stream processing languages Jon Riecke • rule oriented languages Adrian Paschke Languages Styles Break • Agent oriented languages Opher Etzion • Temporal semantics Jon Riecke Advanced • Integrated development environments Jon Riecke Topics • Formal approaches Adrian Paschke • Conclusion Opher Etzion 12
  • 13. epts event processing technical society Hallmarks of Stream Processing • Events are processed in a directed graph • Nodes = computational elements • Edges = communication of events between nodes • Languages inspired by SQL/relational algebra (though not all use an extended SQL as the language) • Difference: write the query first • Results of query are updated as data flows in • Academic examples: STREAM, Aurora, Cayuga, … • Commercial examples: Aleri, Coral8, Esper/Nesper, StreamBase, System S, … 13
  • 14. epts event processing technical society Some pictures 14
  • 15. epts event processing technical society Some pictures 15
  • 16. epts event processing technical society Some pictures 16
  • 17. epts event processing technical society Historical Roots • Relational Databases • Triggers: change table when another table changes • Materialized view maintenance: calculate small set of changes to view given set of changes to base tables • Signal processing • Control and Audio: Ptolemy, LabVIEW, … • Music and video: OpenMusic, jMax, Max, vvvv, 17
  • 18. epts event processing technical society Historical Roots: Static Dataflow Architecture • Developed by Jack Dennis in 1960s • Each arc carries one token • Node fires when all data available on all arcs 18
  • 19. epts event processing technical society Historical Roots: Dynamic Dataflow Architecture • Matching section in front of each node • Tokens are tagged by invocation instance; when all tokens with same tag are ready, node fires • Allows more parallelism than static model 19
  • 20. epts event processing technical society Kahn Dataflow Networks • Non-blocking write/blocking read queues • Deterministic nodes implies deterministic network 20
  • 21. epts event processing technical society Stream Processing versus Signal Processing and Dataflow Architecture • Events carry more information than simple data: records instead of, e.g., decimal numbers • Slightly slower data rates (up to millions of events per second) • Publish/subscribe model of input/output 21
  • 22. epts event processing technical society Dimension Commonalities in Stream Processing • Events: Non-mutable records (fields with values) Values have scalar types (integer, float, string, blob, xml) • Metadata: describes streams and types of records • State: Unprocessed and processed events, and internal state of operations (e.g., aggregations) • Computational model: records traverse the dataflow graph • Programming model: extensions of SQL/relational algebra 22
  • 23. epts event processing technical society Dimension Differences • Events: small differences in scalar types • State • User-defined data structures (arrays, maps, …)? • Disk-based persistence? What is persisted? • Programming model • Visual programming or text-based? • Embedded in larger language, e.g., Java? • Smaller language embedded in it? 23
  • 24. epts event processing technical society Dimension Differences • Computational model • Cycles in dataflow graph? • Deterministic? • Consistent? What form of consistency? • Parallel or concurrent execution? • Distributed computation? Distributed data? • Synchronous or asynchronous communication? 24
  • 25. epts event processing technical society Aleri Streaming Platform Dataflow Networks • One queue per node (instead of per edge) • Nodes store records in table • Events (insert/update/delete) change the table • Nodes run in separate threads • Optional persistence with roll-forward recovery 25
  • 26. epts event processing technical society Aleri Streaming Platform Dataflow Networks • Computational model • Events may arrive in different orders, and may cause different output events • Correctness: standard operators (join, compute, …) satisfy “eventual consistency”: the sequence of insert/update/deletes will produce the same table, no matter how the events flow through the graph 26
  • 27. epts event processing technical society Aleri Streaming Platform Dataflow Networks B A C • Node A sends insert:[Id=3, Symbol=IBM, Price=45.0] Node B sends update: [Symbol=IBM, AvgPrice=43.40] • Different arrival orders can cause different events • (A then B) • insert: [Id=3, Symbol=IBM, Price=45.0, AvgPrice=42.00] • update: [Id=3, Symbol=IBM, Price=45.0, AvgPrice=43.40] • (B then A) • insert: [Id=3, Symbol=IBM, Price=45.0, AvgPrice=43.40] 27
  • 28. epts event processing technical society Coral8 Dataflow Networks • No queues; synchronization buffer in front • Two kinds of nodes: • Stream nodes process records but don’t store them • Window nodes process and store records • Optional persistence for unprocessed events, records 28 in windows, aggregate state
  • 29. epts event processing technical society Coral8 Dataflow Networks • Computational model (simplified) • External data flows into synchronization buffer • Events carry a timestamp (point in time, microsecond granularity): set on arrival, or by source • Events with same timestamp are fed into the network in one step, and are processed until network quiesces 29
  • 30. epts event processing technical society Coral8 Dataflow Networks • Computational model • Deterministic: CCL programs produce the same output events given the same input • Priorities assigned to nodes in graph to maintain determinacy 30
  • 31. epts event processing technical society Coral8 Dataflow Networks • Computational model: concurrent/distributed • Project = group of CCL streams, windows, queries (code for calculating new events from old) • Projects can send/receive events to/from other projects • Each project has a thread (there are other threads too) 31 • Projects can be distributed across machines
  • 32. epts event processing technical society Examples in CCL: Simple Market Data • Schema: type of records – CREATE SCHEMA Trades_t – (Symbol STRING, Qty INTEGER, Price FLOAT); – CREATE SCHEMA Book_t – (Symbol STRING, Qty INTEGER, BuyPrice FLOAT); • Other scalar types: BOOLEAN, TIMESTAMP, INTERVAL, XML, BLOB • CCL streams: INPUT, OUTPUT, local – CREATE INPUT STREAM Trades_s SCHEMA Trades_t; – CREATE INPUT STREAM Book_s SCHEMA Book_t; 32
  • 33. epts event processing technical society CCL Windows • Windows are like streams, but store records – CREATE WINDOW Book_w SCHEMA Book_t KEEP ALL; – INSERT INTO Book_w – SELECT * FROM Book_s; • Sample of KEEP policies (there are others): – KEEP LAST PER Id – KEEP 3 MINUTES – KEEP EVERY 3 MINUTES – KEEP UNTIL (”MON 17:00:00”) – KEEP 10 ROWS – KEEP LAST ROW – KEEP 10 ROWS PER Symbol 33
  • 34. epts event processing technical society CCL Queries: SQL-based syntax • Aggregation – CREATE STREAM Vwap_s SCHEMA (Symbol STRING, Vwap FLOAT); – INSERT INTO Vwap_s – SELECT Symbol, sum(Qty * Price)/sum(Qty) – FROM Trades_s KEEP 30 MINUTES – GROUP BY Symbol; • Join – CREATE SCHEMA Value_t – INHERITS FROM Book_t (CurVal FLOAT, AvgVal FLOAT); – CREATE WINDOW BookValue_w – SCHEMA Value_t KEEP LAST PER Symbol; – INSERT INTO BookValue_w – SELECT B.Symbol, B.SharesHeld, B.BuyPrice, – B.SharesHeld * L.Price, B.SharesHeld*V.Vwap – FROM Book_w B, LastTrade_w L, Vwap_s V – WHERE B.Symbol = L.Symbol and B.Symbol = V.Symbol; 34
  • 35. epts event processing technical society CCL Pattern Matching • MATCHING clause – INSERT INTO OrdersMatch_s – SELECT B.ID, S.ID, B.Price – FROM BuyOrders_s B, Trades_s T, SellOrders_s S, – MATCHING [30 SECONDS: B, !T, S] – ON B.Price = S.Price = T.Price; • Operators in patterns (cf. finite-state automata): • Three boolean operators: ! (not), & (and), | (or) • One temporal operator: , (followed by) • Temporal scope can be nested – MATCHING [30 SECONDS: [10 SECONDS: B, !T], S] 35
  • 36. epts event processing technical society Modularity • Note: concrete syntax in future version of CCL • Group of streams can be abstracted into a module – CREATE MODULE OptionPrice – CREATE INPUT STREAM Option_s; – CREATE OUTPUT STREAM OptionPrice_s; – CREATE PARAMETER FLOAT InterestRate = 0.05; – // internal stream definitions – END MODULE; • Modules can be instantiated ● LOAD MODULE OptionPrice as LondonOptionPrices ● STREAMS ● Input = LondonOptions_s, ● Output = LondonOptionPrices_s ● PARAMETERS ● InterestRate = 0.03; 36
  • 37. epts event processing technical society Other Features from Other Languages • Esper (Java)/Nesper (C#) • Extensions of SQL • Embedded within host language • AleriML • SPLASH: a C-like embedded language for writing nodes • Data structures: vectors, dictionaries, event caches • Mechanisms for discarding stored records (time, # records) • StreamBase • Primitive types include nested records and vectors (lists) • LOCK/UNLOCK for holding and releasing records 37
  • 38. epts event processing technical society TUTORIAL OUTLINE Introduction Opher Etzion stream processing languages Jon Riecke Rule oriented languages Adrian Paschke Languages Styles • Break • Agent oriented languages Opher Etzion • Temporal semantics Jon Riecke Advanced • Integrated development environments Jon Riecke Topics • Formal approaches Adrian Paschke • Conclusion Opher Etzion 38
  • 39. epts event processing technical society Production Rules 39
  • 40. epts event processing technical society Production Rules and CEP • Production rules react to states changes (not events) • PR systems with object model and external fact updates might be extended to CEP – Event types and classes are defined in the rule declarations – New event data instances are added to the fact base / working memory • might be an external fact base, e.g. a event queue – The instances of declarations are filtered and joined in the conditions via typed pattern matching and if they pass the condition list the action part of the rule is triggered • Might be further extended with mechanisms like query languages, state models and temporal constraints • Examples: TIBCO Business Events, Drools 40
  • 41. epts event processing technical society Production Rules • A production rules is a statement of the form: “if Condition then Action” • Members, e.g. OPS5, Clips, Jess, Drools, Fair Isaac Blaze Advisor, ILog jRules, CA Aion, Haley, ESI Logist, … • Operational semantics: forward-chaining – primitive update actions (assert, retract, …); 41
  • 42. epts event processing technical society Operational Semantics • Operational processing – forward-chaining order-independent execution (e.g. based on variations of the RETE algorithm) – Procedural order-dependent sequential execution in a (compiled) execution environment (rule engine). • Conflict resolution – Refraction – Priority – Recency – Specificity – … 42
  • 43. epts event processing technical society The Rete Pattern-Matching Algorithm • Rete Pattern-Matching Algorithm (and variations of it) – matches facts against the patterns in rules to determine which rule conditions are satisfied – incremental evaluation of production rules' conditions • Developed by Charles L. Forgy in the late 70s for OPS5 (Official Production System) shell – stores information about the antecedents in a network – in every cycle, it only checks for changes in the networks  this greatly improves efficiency 43
  • 44. epts event processing technical society OMG Production Rules Representation (OMG PRR) • Formal model for vendor-neutral rule-model representation in UML for production rules • Consortium of developers and supporters – Rule vendors (including Fair Isaac, IBM/ILOG, LibRT, Pega, Corticon, TIBCO) – Academic community (RuleML Group) – Related vendor community (Fujitsu) • PRR is currently “adopted" as a standard and in “finalization", meaning that it is in Beta, with a final version 1.0 in 2009. 44
  • 45. epts event processing technical society OMG Production Rules Representation • OMG MDA PIM model • PRR beta spec applies to a PRR Core using OMG MOF defined in UML – Extends UML so production rules are 1st class citizens alongside objects – Spec. excludes an explicit expression language • Can use other standards as expression language – e.g. W3C Rule Interchange Format (W3C RIF) and the production rule language of RuleML 45
  • 46. epts event processing technical society OMG PRR ProductionRule Classes 46 if [condition] then [action]
  • 47. epts event processing technical society W3C Rule Interchange Format • W3C Rule Interchange Format (W3C RIF) http://www.w3.org/2005/rules/wiki/RIF_Working_Group • Goal Standardization of a web-based Rule Interchange Format (RIF) • RIF 1.0 three dialects: – RIF Production Rules Dialect (RIF-PRD) • specifies the RIF production rules dialect to enable the interchange of production rules – RIF Basic Logic Dialect (RIF-BLD) • specifies a basic interchange format that allows logic rules (definite Horn rules with equality) to be exchanged – RIF Core • RIF-Core, a common subset of RIF-BLD and RIF-PRD 47
  • 48. epts event processing technical society W3C RIF Production Rules Dialect • Production Rules (Condition-Action) – Based on RIF condition language (from RIF Core) – Actions: Retract, Assert, Modify, Execute – Negation (inflationary negation) • Syntax – Normative XML syntax • Might be concrete representation language for OMG PRR – Non-normative EBNF-based presentation syntax • for presentation purposes 48
  • 49. epts event processing technical society RIF Production Rule <Implies> <if> FORMULA </if> <then rif:ordered="yes"> <Do> ACTION* </Do> </then> Syntax close to PR RuleML syntax </Implies> 49
  • 50. epts event processing technical society Event Condition Action Rules 50
  • 51. epts event processing technical society Event Condition Action Rules • ECA Rule “on Event if Condition do Action”; – Explicit complex event part; separated from conditions and actions • e.g. expresses customer order (event); check if credit card is valid (condition) • Evolved from active databases – extend databases with reactions, e.g. HiPac, ACCOOD, Chimera, ADL, COMPOSE, NAOS – Composite event algebra, e.g. SAMOS, COMPOSE, Snoop • Sequence | Disjunction | Xor | Conjunction | Concurrent | Not | Any | Aperiodic | Periodic 51
  • 52. epts event processing technical society Time-point vs. Interval-based Semantics – Time-point semantics • Event definition: B; (A;C) (Sequence) event instance sequence EIS = b, a, c => detect event event instance sequence EIS = a, b, c => detect event – Interval-based semantics • Event definition B;(A;C) (Sequence) event instance sequence EIS = b, a, c => detect event event instance sequence EIS = a, b, c => do not detect event T1 T2 T3 T4 B A C 52
  • 53. epts event processing technical society Operational Semantics 1. Specification Phase – Definition of event/action pattern e.g. by event algebra – Based on declarative formalization or procedural implementation – Defined over an atomic instant or an interval of time, events/actions, situation, transition etc. 2. Selection – Defines selection function to select one event from several occurred events of a particular type in an event instance sequence, e.g. “first”, “last” – Crucial for the outcome of a reaction rule, since the events may contain different (context) information, e.g. different message payloads or sensing information 53
  • 54. epts event processing technical society Operational Semantics 1. Consumption – Optionally defines which events are consumed after the detection of a composite event – An event may contribute to the detection of several composite events, if it is not consumed – Distinction in event messaging between “multiple receive” and “single receive” – Events which can no longer contribute, e.g. are outdated, should be removed 2. Execution – Actions might have an internal effect i.e. change the knowledge state leading to state transition from (pre)-condition state to post-condition state. – Actions might have an external side effect 54
  • 55. epts event processing technical society Example: REWERSE MARS • MARS (Modular Active Rules for the Semantic Web) – rule uses specialized heterogeneous event language (E), one or more query languages (Q), a test language (T), and an action language (A) – each of these languages and their constructs are described by metadata and an ontology of its semantics – well-defined interface for communication between the E, Q&T, and A components by variables – SNOOP (E), OWLQ (Q&T) and CCS - Calculus of Communicating Systems (A) 55
  • 56. epts event processing technical society Example: XChangeEQ • ECA rules ON Event query IF Web query DO Action – the condition part is a declarative query to XML data – the event part is as well a declarative query to XML event data • distributed and web-based: events issued by web nodes (with both emission and reception times) – - action: simple or complex updates send to web nodes • Complex events are expressed as (compound) queries to the event stream – temporal restrictions and event compositions • conjunction, disjunction, exclusion, quantifications, repetitions, and ranks • absolute temporal restrictions, also relative temporal restrictions (within) 56
  • 57. epts event processing technical society XChangeEQ • The four dimensions of event querying are kept separated: – data extraction (eg oder number, book title) – event composition (eg oder of a dictionary and a grammar both from the same publisher) – temporal (and other) relationships (eg two orders within a week) – event accumulation (eg no default of paiement between the order of the two abovementioned buys) • Equivalent Semantics: – Declarative semantics: Model and fixpoint theories – Operational semantics: CERA (Complex Event Relational Algebra) • XChange^EQ operational semantics: – by language design, the system never wait forever for a complex event to possibly resume. – Incremental evaluation (refining RETE). – Built-in garbage collection of irrelevant complex events (eg events that cannot resume). 57
  • 58. epts event processing technical society Example: XChange Composite Event andthen [ xchange:event {{ xchange:sender {"http://airline.com"}, cancellation-notification {{ flight {{ number { var Number } }} }} }}, xchange:event {{ xchange:sender {"http://airline.com"}, important {"Accomodation is not granted!"} }} ] within 2 h 58
  • 59. epts event processing technical society ruleCore Markup Language (rCML) • XML language for ECA rules • Complex Event defined in four sub-elements: – <detect-event>. The <detect-event> element specifies the event that is generated when the composite event is detected. – <no-detect-event>. The <no-detect-event> element specifies the event that is generated when the composite event can not be detected. – <event-selector> The <event-selector> element specifies a logical condition (or filter) for the composite event. – <detector>. The composite event detector consists of a number of event algebra operators (the SNOOP algebra operators) that describe the composite event. 59
  • 60. epts event processing technical society IBM AMIT Situation Manager Rule Language • Definition of situations and their detection conditions – syntactically equivalent to (complex) event patterns – lifespans which hold references to events with relevant context for a particular pattern calculation • Conceptual event model defines an event type generalization hierarchy – Root event type defines a set of standard attributes e.g., • the event source, creation type, temporal dimensions like occurrence time, detection time and /or transaction time of an event, and quantifications such as status, priority, severity, count – Application specific event types inherit from root event type and define further user-defined attributes – Two types of event relationships • Based-on - a pure syntactical relationship to ease the development process. If an event A is 'basedon' event B, event A inherits the entire structure of event B and can extend it. • Reference - a relationship that allows deriving events without pattern detection. Useful for transformation and enrichment purposes. 60
  • 61. epts event processing technical society Example AMIT SMRL AMIT SMRL markup language provides an event algebra with semantic notations such as context (lifespan intervals), and semantic association (event group) to define situations <eventTypes> <eventType name="doctorEnterRoom"> <attributeType name="doctor" xsi:type="string"/> <attributeType name="room" xsi:type="integer"/> </eventType> <eventType name="patientEnterRoom"> <attributeType name="patient" xsi:type="string"/> <attributeType name="r_o_o_m" xsi:type="integer"/> </eventType> <eventType name="lockRoom"> <attributeType name="room" xsi:type="integer"/> </eventType> <eventType name="unlockRoom"> <attributeType name="room" xsi:type="integer"/> </eventType> </eventTypes> <lifespan name="roomOpen"> <initiator> <eventInitiator name="unlockRoom"/> </initiator> <terminator> <eventTerminator name="lockRoom"/> </terminator> <keyBy name="sameRoom"/> </lifespan> 61
  • 62. epts event processing technical society (Reaction) RuleML • General (reaction) rule form that can be specialized as needed • Platform-independent XML-based rule interchange format – translation into platform-specific executable rule languages, e.g. Prova • Three general execution styles: – Active: 'actively' polls/detects occurred events in global ECA style, e.g. by a ping on a service/system or a query on an internal or external event database – Messaging: Waits for incoming complex event message – Reasoning: KR event/action logic reasoning and transitions (as e.g. in Event Calculus, Situation Calculus, TAL formalizations) • Appearance – Global: ‘globally’ defined reaction rule – Local: ‘locally’ defined reaction rule in a specific context 62
  • 63. epts event processing technical society General Syntax for Reaction Rules <Rule style="active" eval="strong"> <on> <!-- event --> </on> <if> <!-- condition --> </if> <do> <!-- action --> </do> <ifPost> <!-- postcondition --> </ifPost> <doAlternative> <!-- alternative/else action --> </doAlternative> </Rule> 63
  • 64. epts event processing technical society Selected Reaction RuleML Extended Features • Action Algebra: Succession (Ordered Succession of Actions), Choice (Non-Determenistic Choice), Flow (Parallel Flow), Loop (Loops) • Event Algebra: Sequence (Ordered), Disjunction (Or) , Xor (Mutal Exclusive), Conjunction (And), Concurrent , Not, Any, Aperiodic, Periodic • Support for event / action messaging • Support for different detection, selection and consumption policies • Support for intervals (Time, Event) • Support for situations (States, Fluents) • Support for external event query languages • ... 64
  • 65. epts event processing technical society 10 minutes break TUTORIAL OUTLINE Introduction Opher Etzion stream processing languages Jon Riecke  rule oriented languages Adrian Paschke Languages Styles Break • Agent oriented languages Opher Etzion • Temporal semantics Jon Riecke Advanced • Integrated development environments Jon Riecke Topics • Formal approaches Adrian Paschke • Conclusion Opher Etzion 65
  • 66. epts event processing technical society TUTORIAL OUTLINE Introduction Opher Etzion stream processing languages Jon Riecke  rule oriented languages Adrian Paschke Languages Styles  Break • agent oriented languages Opher Etzion • Temporal semantics Jon Riecke Advanced • Integrated development environments Jon Riecke Topics • Formal approaches Adrian Paschke • Conclusion Opher Etzion 66
  • 67. epts event processing technical society EPA and EPN • In David Luckham’s “the power of events” the term EPN (Event Processing Network) was used as a collection of EPAs (Event Processing Agents). • The EPN describes the c1 p1 admittance Filter: Diabetes p5 c5 EPN Example “programming in the large”, Pattern detection: Alert physician p6 p7 p12 while each individual agent Blood pressure reading p8 Enrich: p10 p11 describes the c2 c6 p2 p9 Is diabetic? p12 Aggregator: Max. “programming in the c7 Fever C p14 Translate: p13 c3 C2F fever p19 Fever F p3 small”. p4 Fever F c8 p15 Pattern Detection: Alert Nurse p16 p20 Pattern Detection: p18 Continuous Fever p17 c9 IBM Haifa Research Lab – Event Processing © 2008 IBM Corporation 67
  • 68. epts event processing technical society Some EPN, event flow and data flow oriented languages Agent Logic Event Zero Oracle/BEA Spade (IBM System S) IBM InfoSphere Streams Streambase A SPADE program builds a data-flow network out of operators source stream processing sink operator operator operator m str ea ea str m processing stream processing operator operator stre m ea am str source sink operator operator 68 194 Tutorial on the SPADE language © 2008, 2009 IBM Corporation
  • 69. epts event processing technical society What is the idea ? • The EPN (event processing network) represents the event processing application as a directed graph: – The nodes represent event processing agents (and states in some models) – The edges represent either individual events or event streams (depends on the processing type). A generalized EPN may support both. • The coverage of EPN varies – It may cover some language operator (e.g. SQL like language) – It may take a broader approach and cover also routing decisions, pub/ sub etc… – EPN can have a specific programming model, or an hybrid programming model, where one or more languages represent the agents. 69
  • 70. epts event processing technical society Anatomy of Agents Performs the processing Selects the events that are Derives output events relevant for the processing detect patterns) (e.g. Selector Processor Deriver 70
  • 71. epts event processing technical society Event Processing Specialized Agent types Event Processing Agent Filter Transform Detect Pattern Route Translate Aggregate Split Compose Enrich Project 71
  • 72. epts event processing technical society Routing – channel EPA Itinerary-based routing Subscription-based routing Intelligent routing 72
  • 73. epts event processing technical society Making the agents context sensitive Time interval that starts by: {event, clock, offset from event} and terminates by: {event, clock, offset from event, count of certain events} – each can have multiple instances (e.g. several events, periodic clock, multiple offsets} When? Examples: [9:00AM, angry-customer-alert], [angry- customer-alert, +1 hour], [request – 2 hours, request] Spatial dimensions/ abstraction Where? Inside building A, In Italy, within 1KM from here (given GPS coordinate) An entity or a collection of entities that determine the partition of the context Who? Class of customer, customer + order, large order… The event happens when some state is in effect Which state? Traffic Jam, Red Alert, Bullish Market 73
  • 74. epts event processing technical society Filtering Many of the raw events are not relevant for processing, in some applications only a sample is selected for processing. 74
  • 75. epts event processing technical society Transformation Translation / projection: Translation and simple derivations Aggregation/ composition: statistical aggregator or concatenating events, can be stand alone agent or a scalar derivation. May be in network edge. Splitting: Splits events to multiple events 75
  • 76. epts event processing technical society Enrichment Enriches the content of events from reference data in databases, spreadsheets, Email messages, text files etc.. 76
  • 77. epts event processing technical society Pattern Matching Pattern definition example In many cases, the agent does not consume raw events, but event that are derived from pattern matching. DEBS 2008 tutorial on event processing patterns provided a deep dive 77
  • 78. epts event processing technical society Dimensions - I • Meta-data – EPN and agent definitions may be defined as meta-data – Event schema is implementation dependent • State – State can be represented as another node in the EPN – Some EPAs may be state oriented, e.g. the input event is change in state – State is one of the context dimensions 78
  • 79. epts event processing technical society Dimensions - II • Execution model: – EPN representation is natural one for concurrency/parallelism since the dependencies are explicit – Since it models explicitly the end-to-end event flow, it serve as a convenient model for QoS related optimizations (latency, throughput, real-time constraints) • Programming model: – Can support multiple programming styles – Can support all types of event processing functionality 79
  • 80. epts event processing technical society p1 p5 c1 c5 Filter: Diabetes admittance p6 EPN example Pattern detection: p12 Alert physician p7 Blood pressure reading p11 p8 p10 c2 Enrich: p2 Is diabetic? c6 p9 p12 Aggregator: Max. c7 Fever C p14 Translate: p13 c3 C2F fever p19 Fever F p3 Fever F p15 Pattern Detection: p4 Alert Nurse c8 p16 p20 Pattern Detection: p18 Continuous Fever p17 c9 80
  • 81. epts event processing technical society Event Zero example 81
  • 82. epts event processing technical society System S Example Congestive Heart Failure Freq alert Arrhythmia Analyzer QRS RR P and T Source PE Glucose alert BP S/D CHF Weight trend IHE detector Adapter *Daby Sow 82
  • 83. epts event processing technical society Agent Logic example 83
  • 84. epts event processing technical society TUTORIAL OUTLINE Introduction Opher Etzion stream processing languages Jon Riecke  rule oriented languages Adrian Paschke Languages Styles  Break agent oriented languages Opher Etzion • Temporal semantics Jon Riecke Advanced • Integrated development environments Jon Riecke Topics • Formal approaches Adrian Paschke • Conclusion Opher Etzion 84
  • 85. epts event processing technical society Varieties of Temporal Semantics • Do events carry timestamps? Where does the timestamp come from (CEP system or external world)? • Granularity? Second, millisecond, microsecond, …? • Point in time or interval of time? • Can events arrive out-of-order from a single data source? • Can events be reordered if they arrive out-of-order? • How long does the system wait? • What happens if system moves ahead, and old event arrives? • In distributed case, is there a difference between emission and reception time? 85
  • 86. epts event processing technical society Simple Temporal Semantics • Events arrive in order from a single data source • Events have timestamps, but not reordered • Advantage • Process as soon as possible • Disadvantages • Non-deterministic • Hard to manage if time really matters • Emission times de-synchronized in distributed case 86
  • 87. epts event processing technical society Global Clock • Events have timestamps consistent across data sources • Wait for events for specified time; discard if wait too long • Advantages • Deterministic • Time might be relevant • Disadvantages • Speed of processing • Buffering required for reordering events 87
  • 88. epts event processing technical society TUTORIAL OUTLINE Introduction Opher Etzion stream processing languages Jon Riecke  rule oriented languages Adrian Paschke Languages Styles  Break agent oriented languages Opher Etzion  Temporal semantics Jon Riecke Advanced • Integrated development environments Jon Riecke Topics • Formal approaches Adrian Paschke • Conclusion Opher Etzion 88
  • 89. epts event processing technical society Integrated Development Environments (IDEs) • Two main forms • Text based • Visual or graphical based • Not an either-or; most systems have some combination of text and visual display of dataflow 89
  • 90. epts event processing technical society Visual IDEs • Streams = nodes, flows = edges • Drag-and-drop streams from palette onto canvas and wire together • Configure streams with property sheets or text • Advantages: simple metaphor; easy to see flow • Disadvantages: managing screen real-estate 90
  • 91. epts event processing technical society Visual IDE Example 91
  • 92. epts event processing technical society Text-based IDEs • Streams represented in text • Advantages • Appeals to programmers • Can be quicker to write code • Disadvantages • Hard to see the data flow • Hard to find bottlenecks in program 92
  • 93. epts event processing technical society Text-based IDE 93
  • 94. epts event processing technical society Another Text-based IDE 94
  • 95. epts event processing technical society Advanced Features in IDEs • Ability to start/stop programs • Debugging, including viewing intermediate events and setting breakpoints • Tests, including ability to control data feeds • Dashboards for displaying results 95
  • 96. epts event processing technical society Breakpoints and Debugging 96
  • 97. epts event processing technical society Performance Monitoring 97
  • 98. epts event processing technical society Dashboard Construction 98
  • 99. epts event processing technical society Dashboard Construction 99
  • 100. epts event processing technical society Assessment: IDEs still have a ways to go • Need more features that programmers have come to expect • Single-stepping with line numbers • Examination of events and data • Forward and backward replay of time • Need to be integrated into more dashboards and other visualization tools 100
  • 101. epts event processing technical society TUTORIAL OUTLINE Introduction Opher Etzion stream processing languages Jon Riecke  rule oriented languages Adrian Paschke Languages Styles  Break agent oriented languages Opher Etzion  Temporal semantics Jon Riecke Advanced  Integrated development environments Jon Riecke Topics • Formal approaches Adrian Paschke • Conclusion Opher Etzion 101
  • 102. epts event processing technical society Formal Approaches Temporal Event/Action Logics for Reasoning on Changes 102
  • 103. epts event processing technical society (Temporal) Event Logics • (Temporal) event/action logics – Events with effects on changeable properties / states / fluents – Focus: reasoning on effects of events/actions on knowledge states and properties – Not Complex Event Processing Languages, but can be used in reasoning on/with complex events • Members e.g. – event calculus and variants, – situation calculus, – features and fluents calculus, – various (temporal) action languages (TAL), – fluent calculi and versatile event logics. 103
  • 104. epts event processing technical society The Situation Calculus • A situation is a snapshot of the world at some point in time • Every true or false statement is made with respect to a particular situation • When an agent performs an action A is situation S1, the result is a new situation S2 • McCarthy 1963, McCarthy & Hayes 1969, Green 1969 104
  • 105. epts event processing technical society Event Calculus • Kowalski and Sergot’s EC – a formalism for temporal reasoning about events and their effects • computation of earlier events (long-term "historical" perspective) – model of change • events happen at time-points and initiate and/or terminate properties (time- varying fluents) of the world – Law of inertia: Things normally tend to stay the same • Main differences in EC to SC – branching time in SC vs. linear time in EC – explicit notion of previous state of the world / situation in SC – state transitions are functions in SC 105
  • 106. epts event processing technical society Classical Event Calculus Example • EC Basic Axioms: – happens(E,T) event E happens at time point T – initiates(E,F,T) event E initiates fluent F for all time>T – terminates(E,F,T) event E terminates fluent F for all time>T – holdsAt(F,T) fluent F holds at time point T • Many EC Extensions, e.g.: – valueAt(P,T,X) parameter P has changeable value X at time point T – planned(E,T) event E is believed to happen at time point T Example: initiates(stopService,serviceUnavailable,T) stopService startService terminates(startService,serviceUnavailable,T) happens(stopService,t1); happens(startService,t5) t1 t3 ? t5 t7 ? holdsAt(serviceUnavailable,t3)?  true holdsAt(serviceUnavailable,t7)?  false 106
  • 107. epts event processing technical society Example: Interval-based Event Calculus Event Algebra – Event initiate and terminate Situations (Fluents) which hold at an time interval – Interval-based Event Calculus semantics (model-theory + proof theory) based on time intervals modeled as fluents I: Ti x Fl  {true, false} – Example: B;(A;C) (Sequence) T1 T2 T3 T4 A B C (A;B;C) [T1,T3]) <= holdsInterval([a,b] [T1,T2]) Λ holdsInterval([b,c] [T2,T3]) Λ [T1,T2]<=[T2,T3] – Rule-based implementation of EC event algebra, e.g. as meta logic program – Rule-based arithmetic involving times and durations, e.g. Allen’s interval logic 107
  • 108. epts event processing technical society Combinations of Rule Types 1. Rules that influence the operational / decision processes: • Derivation rules (deduction rules): establish / derive new information that is used e.g. in a decision process. • Reaction rules that establish when certain activities should place (e.g. ECA rules) 2. Constraints on system/organisation's structure, behavior or information: • Structural constraints (e.g. deontic assignments), State constraints, Process / flow constraints 108
  • 109. epts event processing technical society Example: Prova Messaging Reaction Rules %Event Message rcvMsg(EventID, jms, Requester, acl_query-ref, Task) :- %Condition(s) – “find available service” available(Service), variable data binding %Action Message – “load task on service” sendMsg(Sub-ID,jms,Service,acl_query-ref, load(Task)). % Derivation rules available(S) :- service(S), not(maintenance(S)), not(loaded(S),T). maintenance(S):- … automatic rule chaining loaded(S) :- … 109
  • 110. epts event processing technical society Messaging Reaction Rules in Prova • Send a message sendMsg(XID,Protocol,Agent,Performative,[Predicate|Args]|Context) • Receive a message rcvMsg(XID,Protocol,Agent,Performative,[Predicate|Args]|Context) • Receive multiple messages rcvMult(XID,Protocol,Agent,Performative,[Predicate|Args]|Context) Syntax: – XID is the conversation / event instance sequence identifier – Protocol: ESB transport protocols (>30 e.g. self, jade, jms, soap,…) – Agent: denotes the target or sender of the message – Performative: pragmatic context, e.g. FIPA ACL primitives – [Predicate|Args] or Predicate(Arg1,..,Argn): Message payload 110
  • 111. epts event processing technical society TUTORIAL OUTLINE Introduction Opher Etzion stream processing languages Jon Riecke  rule oriented languages Adrian Paschke Languages Styles  Break agent oriented languages Opher Etzion  Temporal semantics Jon Riecke Advanced  Integrated development environments Jon Riecke Topics  Formal approaches Adrian Paschke Conclusion Opher Etzion 111
  • 112. epts event processing technical society Event Processing Languages – insights from this tutorial epts event processing technical society epts event processing technical society An Observation Dimensions toTower symbolizeslanguages The Babylon describe the tendency Of humanity to talk in multiple languages. • Events Data and Meta-data • State • no different: most languages in the model The Event Processing area is Computation/execution industry really follow the hammer and nails syndrome – and extended existing approaches • Programming model • imperative script language • SQL extensions • Extension of inference rule language It does not seem that we’ll succeed to settle In the near future around a single programming style The epts language analysis workgroup is aimed to understand the various styles And extract common functions that can be used to define what is an event processing language; this tutorial is an interim report 112
  • 113. epts event processing technical society Towards event processing languages standard ? • Eventually – a single language ? 1st generation SQL standard approved of relational DBMS products Mid 1970-ies 1990 1st generation EP standard approved of EP products Mid 200-ies ? • OR – variety of language style standards ? • Meta-language as an intermediate standard ? 113
  • 114. epts event processing technical society The epts “Event Processing Language Analysis” workgroup Mission: The purpose of this working group is to conduct a study of the features that exist in the contemporary and planned languages in the area of event processing (including event stream processing). The goal is to understand the functional properties that exist in event processing languages and abstract out semantic functions, regardless of syntax and implementation considerations. This is a first step in a way to determine future possible standardization in the event processing area WG Leaders: Opher Etzion (IBM) and Jon Riecke (Aleri) Members: • Alex Kozlenkov (Betfair) • Adrian Paschke (FU Berlin) • Arno Jacobsen (U of Toronto) • Bala Maniymaran (U of Toronto) • Bob Hagmann (Aleri) • David Tucker (EventZero) • Dermot McPeake (FirstDerivatives) • Francois Bry (LMU) • Guy Sharon (IBM) • Louis Lovas (Progress/Apama) • Michael Eckert (TIBCO) • Pedro Bizarro (University of Coimbra) • Richard Tibbetts (Streambase) • Robert McKeown (IBM) • Susan Urban (Texas Tech. University) • Simon Courtenage (Westminister U.) • Serge Mankovskii (CA) 114
  • 115. epts event processing technical society If you want to participate in these discussions, contribute to this topic and/or others topics, and be part of the community effort to advance the state of the art and state of the practice: JOIN EPTS For details: http://www.ep-ts.com/