Skip to main content
UploadLanguage (EN)Support
BusinessMobileSocial MediaMarketingTechnologyArt & PhotosCareerDesignEducationPresentations & Public SpeakingGovernment & NonprofitHealthcareInternetLawLeadership & ManagementAutomotiveEngineeringSoftwareRecruiting & HRRetailSalesServicesScienceSmall Business & EntrepreneurshipFoodEnvironmentEconomy & FinanceData & AnalyticsInvestor RelationsSportsSpiritualNews & PoliticsTravelSelf ImprovementReal EstateEntertainment & HumorHealth & MedicineDevices & HardwareLifestyle
Opher Etzion, profile picture
Uploaded byOpher Etzion
PPT, PDF6,349 views

Debs2009 Event Processing Languages Tutorial

The document outlines a tutorial on event processing languages. It discusses different styles of event processing languages including stream processing languages, rule oriented languages, and agent oriented languages. For stream processing languages, it covers key concepts like events, state, computational models, and programming models. It provides examples of stream processing languages like Esper, Coral8, and CCL.

Technology◦Education◦
Downloaded 845 times
epts event processing technical society Event Processing Language Tutorial François Bry, Michael Eckert, Opher Etzion, Adrian Paschke, Jon Riecke on behalf of the epts languages analysis working group
epts event processing technical society TUTORIAL OUTLINE • Introduction Opher Etzion • stream processing languages Jon Riecke • rule oriented languages Adrian Paschke Languages Styles Break • Agent oriented languages Opher Etzion • Temporal semantics Jon Riecke Advanced • Integrated development environments Jon Riecke Topics • Formal approaches Adrian Paschke • Conclusion Opher Etzion 2
epts event processing technical society TUTORIAL OUTLINE • Introduction Opher Etzion • stream processing languages Jon Riecke • rule oriented languages Adrian Paschke Languages Styles Break • Agent oriented languages Opher Etzion • Temporal semantics Jon Riecke Advanced • Integrated development environments Jon Riecke Topics • Formal approaches Adrian Paschke • Conclusion Opher Etzion 3
epts event processing technical society What is it all about? Filter, transform, enrich, route, Detect patterns, derive events Event Processing Event Event Producer Consumer Generate and emit Consume and react events perform operations on to either raw or events complex events Event processing has gained a lot of interest in recent years; according to analysts it is the fastest growing segment of middleware software In this tutorial we are Pattern exploring the language Match issue – how the various event processing functions are expressed? 4
epts event processing technical society An Observation The Babylon Tower symbolizes the tendency Of humanity to talk in multiple languages. The Event Processing area is no different: most languages in the industry really follow the hammer and nails syndrome – and extended existing approaches • imperative script language • SQL extensions • Extension of inference rule language It does not seem that we’ll succeed to settle In the near future around a single programming style The epts language analysis workgroup is aimed to understand the various styles And extract common functions that can be used to define what is an event processing language; this tutorial is an interim report 5
epts event processing technical society Existing Styles for EP languages (samples) Inference TIBCO Rules Prova WBE XChangeEQ ECA Rules RuleCore AMiT Starview Agent Agent Logic Oriented EventZero Spade SQL Aleri extension Streambase Coral8 State Oracle oriented Esper Imperative/ Netcool Impact Script Based Apama * - if we add simple and mediated event processing the picture is even more diversified 6
epts event processing technical society Dimensions to describe languages • Events Data and Meta-data • State • Computation/execution model • Programming model 7
epts event processing technical society The event Data and meta-data dimension • Event structures supported (structured, semi- structured, unstructured) • Data types supported for event’s payload • Event identifiers/identities • Event time-stamps and ordering • Event relations • Other meta-data entities in the language 8
epts event processing technical society The state dimension • State of events / data – Events and other data in states – Life-cycle, duration policies – Explicit visibility and manipulation of states • State of execution – Context-related grouping (e.g. time windows) – Explicit visibility into state of execution 9
epts event processing technical society The execution model dimension • Execution timing – immediate or deferred ? • Guarantees: real-time, determinism, latency ? • Time point or time interval semantics • Concurrency – implicit / explicit • Event at a time / set at a time processing • Explicit event flow 10
epts event processing technical society The programming model dimension • Programming style • Abstractions • Pattern matching • Enrichment capabilities • Transformation capabilities • Filter capabilities 11
epts event processing technical society TUTORIAL OUTLINE Introduction Opher Etzion stream processing languages Jon Riecke • rule oriented languages Adrian Paschke Languages Styles Break • Agent oriented languages Opher Etzion • Temporal semantics Jon Riecke Advanced • Integrated development environments Jon Riecke Topics • Formal approaches Adrian Paschke • Conclusion Opher Etzion 12
epts event processing technical society Hallmarks of Stream Processing • Events are processed in a directed graph • Nodes = computational elements • Edges = communication of events between nodes • Languages inspired by SQL/relational algebra (though not all use an extended SQL as the language) • Difference: write the query first • Results of query are updated as data flows in • Academic examples: STREAM, Aurora, Cayuga, … • Commercial examples: Aleri, Coral8, Esper/Nesper, StreamBase, System S, … 13
epts event processing technical society Some pictures 14
epts event processing technical society Some pictures 15
epts event processing technical society Some pictures 16
epts event processing technical society Historical Roots • Relational Databases • Triggers: change table when another table changes • Materialized view maintenance: calculate small set of changes to view given set of changes to base tables • Signal processing • Control and Audio: Ptolemy, LabVIEW, … • Music and video: OpenMusic, jMax, Max, vvvv, 17
epts event processing technical society Historical Roots: Static Dataflow Architecture • Developed by Jack Dennis in 1960s • Each arc carries one token • Node fires when all data available on all arcs 18
epts event processing technical society Historical Roots: Dynamic Dataflow Architecture • Matching section in front of each node • Tokens are tagged by invocation instance; when all tokens with same tag are ready, node fires • Allows more parallelism than static model 19
epts event processing technical society Kahn Dataflow Networks • Non-blocking write/blocking read queues • Deterministic nodes implies deterministic network 20
epts event processing technical society Stream Processing versus Signal Processing and Dataflow Architecture • Events carry more information than simple data: records instead of, e.g., decimal numbers • Slightly slower data rates (up to millions of events per second) • Publish/subscribe model of input/output 21
epts event processing technical society Dimension Commonalities in Stream Processing • Events: Non-mutable records (fields with values) Values have scalar types (integer, float, string, blob, xml) • Metadata: describes streams and types of records • State: Unprocessed and processed events, and internal state of operations (e.g., aggregations) • Computational model: records traverse the dataflow graph • Programming model: extensions of SQL/relational algebra 22
epts event processing technical society Dimension Differences • Events: small differences in scalar types • State • User-defined data structures (arrays, maps, …)? • Disk-based persistence? What is persisted? • Programming model • Visual programming or text-based? • Embedded in larger language, e.g., Java? • Smaller language embedded in it? 23
epts event processing technical society Dimension Differences • Computational model • Cycles in dataflow graph? • Deterministic? • Consistent? What form of consistency? • Parallel or concurrent execution? • Distributed computation? Distributed data? • Synchronous or asynchronous communication? 24
epts event processing technical society Aleri Streaming Platform Dataflow Networks • One queue per node (instead of per edge) • Nodes store records in table • Events (insert/update/delete) change the table • Nodes run in separate threads • Optional persistence with roll-forward recovery 25
epts event processing technical society Aleri Streaming Platform Dataflow Networks • Computational model • Events may arrive in different orders, and may cause different output events • Correctness: standard operators (join, compute, …) satisfy “eventual consistency”: the sequence of insert/update/deletes will produce the same table, no matter how the events flow through the graph 26
epts event processing technical society Aleri Streaming Platform Dataflow Networks B A C • Node A sends insert:[Id=3, Symbol=IBM, Price=45.0] Node B sends update: [Symbol=IBM, AvgPrice=43.40] • Different arrival orders can cause different events • (A then B) • insert: [Id=3, Symbol=IBM, Price=45.0, AvgPrice=42.00] • update: [Id=3, Symbol=IBM, Price=45.0, AvgPrice=43.40] • (B then A) • insert: [Id=3, Symbol=IBM, Price=45.0, AvgPrice=43.40] 27
epts event processing technical society Coral8 Dataflow Networks • No queues; synchronization buffer in front • Two kinds of nodes: • Stream nodes process records but don’t store them • Window nodes process and store records • Optional persistence for unprocessed events, records 28 in windows, aggregate state
epts event processing technical society Coral8 Dataflow Networks • Computational model (simplified) • External data flows into synchronization buffer • Events carry a timestamp (point in time, microsecond granularity): set on arrival, or by source • Events with same timestamp are fed into the network in one step, and are processed until network quiesces 29
epts event processing technical society Coral8 Dataflow Networks • Computational model • Deterministic: CCL programs produce the same output events given the same input • Priorities assigned to nodes in graph to maintain determinacy 30
epts event processing technical society Coral8 Dataflow Networks • Computational model: concurrent/distributed • Project = group of CCL streams, windows, queries (code for calculating new events from old) • Projects can send/receive events to/from other projects • Each project has a thread (there are other threads too) 31 • Projects can be distributed across machines
epts event processing technical society Examples in CCL: Simple Market Data • Schema: type of records – CREATE SCHEMA Trades_t – (Symbol STRING, Qty INTEGER, Price FLOAT); – CREATE SCHEMA Book_t – (Symbol STRING, Qty INTEGER, BuyPrice FLOAT); • Other scalar types: BOOLEAN, TIMESTAMP, INTERVAL, XML, BLOB • CCL streams: INPUT, OUTPUT, local – CREATE INPUT STREAM Trades_s SCHEMA Trades_t; – CREATE INPUT STREAM Book_s SCHEMA Book_t; 32
epts event processing technical society CCL Windows • Windows are like streams, but store records – CREATE WINDOW Book_w SCHEMA Book_t KEEP ALL; – INSERT INTO Book_w – SELECT * FROM Book_s; • Sample of KEEP policies (there are others): – KEEP LAST PER Id – KEEP 3 MINUTES – KEEP EVERY 3 MINUTES – KEEP UNTIL (”MON 17:00:00”) – KEEP 10 ROWS – KEEP LAST ROW – KEEP 10 ROWS PER Symbol 33
epts event processing technical society CCL Queries: SQL-based syntax • Aggregation – CREATE STREAM Vwap_s SCHEMA (Symbol STRING, Vwap FLOAT); – INSERT INTO Vwap_s – SELECT Symbol, sum(Qty * Price)/sum(Qty) – FROM Trades_s KEEP 30 MINUTES – GROUP BY Symbol; • Join – CREATE SCHEMA Value_t – INHERITS FROM Book_t (CurVal FLOAT, AvgVal FLOAT); – CREATE WINDOW BookValue_w – SCHEMA Value_t KEEP LAST PER Symbol; – INSERT INTO BookValue_w – SELECT B.Symbol, B.SharesHeld, B.BuyPrice, – B.SharesHeld * L.Price, B.SharesHeld*V.Vwap – FROM Book_w B, LastTrade_w L, Vwap_s V – WHERE B.Symbol = L.Symbol and B.Symbol = V.Symbol; 34
epts event processing technical society CCL Pattern Matching • MATCHING clause – INSERT INTO OrdersMatch_s – SELECT B.ID, S.ID, B.Price – FROM BuyOrders_s B, Trades_s T, SellOrders_s S, – MATCHING [30 SECONDS: B, !T, S] – ON B.Price = S.Price = T.Price; • Operators in patterns (cf. finite-state automata): • Three boolean operators: ! (not), & (and), | (or) • One temporal operator: , (followed by) • Temporal scope can be nested – MATCHING [30 SECONDS: [10 SECONDS: B, !T], S] 35
epts event processing technical society Modularity • Note: concrete syntax in future version of CCL • Group of streams can be abstracted into a module – CREATE MODULE OptionPrice – CREATE INPUT STREAM Option_s; – CREATE OUTPUT STREAM OptionPrice_s; – CREATE PARAMETER FLOAT InterestRate = 0.05; – // internal stream definitions – END MODULE; • Modules can be instantiated ● LOAD MODULE OptionPrice as LondonOptionPrices ● STREAMS ● Input = LondonOptions_s, ● Output = LondonOptionPrices_s ● PARAMETERS ● InterestRate = 0.03; 36
epts event processing technical society Other Features from Other Languages • Esper (Java)/Nesper (C#) • Extensions of SQL • Embedded within host language • AleriML • SPLASH: a C-like embedded language for writing nodes • Data structures: vectors, dictionaries, event caches • Mechanisms for discarding stored records (time, # records) • StreamBase • Primitive types include nested records and vectors (lists) • LOCK/UNLOCK for holding and releasing records 37
epts event processing technical society TUTORIAL OUTLINE Introduction Opher Etzion stream processing languages Jon Riecke Rule oriented languages Adrian Paschke Languages Styles • Break • Agent oriented languages Opher Etzion • Temporal semantics Jon Riecke Advanced • Integrated development environments Jon Riecke Topics • Formal approaches Adrian Paschke • Conclusion Opher Etzion 38
epts event processing technical society Production Rules 39
epts event processing technical society Production Rules and CEP • Production rules react to states changes (not events) • PR systems with object model and external fact updates might be extended to CEP – Event types and classes are defined in the rule declarations – New event data instances are added to the fact base / working memory • might be an external fact base, e.g. a event queue – The instances of declarations are filtered and joined in the conditions via typed pattern matching and if they pass the condition list the action part of the rule is triggered • Might be further extended with mechanisms like query languages, state models and temporal constraints • Examples: TIBCO Business Events, Drools 40
epts event processing technical society Production Rules • A production rules is a statement of the form: “if Condition then Action” • Members, e.g. OPS5, Clips, Jess, Drools, Fair Isaac Blaze Advisor, ILog jRules, CA Aion, Haley, ESI Logist, … • Operational semantics: forward-chaining – primitive update actions (assert, retract, …); 41
epts event processing technical society Operational Semantics • Operational processing – forward-chaining order-independent execution (e.g. based on variations of the RETE algorithm) – Procedural order-dependent sequential execution in a (compiled) execution environment (rule engine). • Conflict resolution – Refraction – Priority – Recency – Specificity – … 42
epts event processing technical society The Rete Pattern-Matching Algorithm • Rete Pattern-Matching Algorithm (and variations of it) – matches facts against the patterns in rules to determine which rule conditions are satisfied – incremental evaluation of production rules' conditions • Developed by Charles L. Forgy in the late 70s for OPS5 (Official Production System) shell – stores information about the antecedents in a network – in every cycle, it only checks for changes in the networks  this greatly improves efficiency 43
epts event processing technical society OMG Production Rules Representation (OMG PRR) • Formal model for vendor-neutral rule-model representation in UML for production rules • Consortium of developers and supporters – Rule vendors (including Fair Isaac, IBM/ILOG, LibRT, Pega, Corticon, TIBCO) – Academic community (RuleML Group) – Related vendor community (Fujitsu) • PRR is currently “adopted" as a standard and in “finalization", meaning that it is in Beta, with a final version 1.0 in 2009. 44
epts event processing technical society OMG Production Rules Representation • OMG MDA PIM model • PRR beta spec applies to a PRR Core using OMG MOF defined in UML – Extends UML so production rules are 1st class citizens alongside objects – Spec. excludes an explicit expression language • Can use other standards as expression language – e.g. W3C Rule Interchange Format (W3C RIF) and the production rule language of RuleML 45
epts event processing technical society OMG PRR ProductionRule Classes 46 if [condition] then [action]
epts event processing technical society W3C Rule Interchange Format • W3C Rule Interchange Format (W3C RIF) http://www.w3.org/2005/rules/wiki/RIF_Working_Group • Goal Standardization of a web-based Rule Interchange Format (RIF) • RIF 1.0 three dialects: – RIF Production Rules Dialect (RIF-PRD) • specifies the RIF production rules dialect to enable the interchange of production rules – RIF Basic Logic Dialect (RIF-BLD) • specifies a basic interchange format that allows logic rules (definite Horn rules with equality) to be exchanged – RIF Core • RIF-Core, a common subset of RIF-BLD and RIF-PRD 47
epts event processing technical society W3C RIF Production Rules Dialect • Production Rules (Condition-Action) – Based on RIF condition language (from RIF Core) – Actions: Retract, Assert, Modify, Execute – Negation (inflationary negation) • Syntax – Normative XML syntax • Might be concrete representation language for OMG PRR – Non-normative EBNF-based presentation syntax • for presentation purposes 48
epts event processing technical society RIF Production Rule <Implies> <if> FORMULA </if> <then rif:ordered="yes"> <Do> ACTION* </Do> </then> Syntax close to PR RuleML syntax </Implies> 49
epts event processing technical society Event Condition Action Rules 50
epts event processing technical society Event Condition Action Rules • ECA Rule “on Event if Condition do Action”; – Explicit complex event part; separated from conditions and actions • e.g. expresses customer order (event); check if credit card is valid (condition) • Evolved from active databases – extend databases with reactions, e.g. HiPac, ACCOOD, Chimera, ADL, COMPOSE, NAOS – Composite event algebra, e.g. SAMOS, COMPOSE, Snoop • Sequence | Disjunction | Xor | Conjunction | Concurrent | Not | Any | Aperiodic | Periodic 51
epts event processing technical society Time-point vs. Interval-based Semantics – Time-point semantics • Event definition: B; (A;C) (Sequence) event instance sequence EIS = b, a, c => detect event event instance sequence EIS = a, b, c => detect event – Interval-based semantics • Event definition B;(A;C) (Sequence) event instance sequence EIS = b, a, c => detect event event instance sequence EIS = a, b, c => do not detect event T1 T2 T3 T4 B A C 52
epts event processing technical society Operational Semantics 1. Specification Phase – Definition of event/action pattern e.g. by event algebra – Based on declarative formalization or procedural implementation – Defined over an atomic instant or an interval of time, events/actions, situation, transition etc. 2. Selection – Defines selection function to select one event from several occurred events of a particular type in an event instance sequence, e.g. “first”, “last” – Crucial for the outcome of a reaction rule, since the events may contain different (context) information, e.g. different message payloads or sensing information 53
epts event processing technical society Operational Semantics 1. Consumption – Optionally defines which events are consumed after the detection of a composite event – An event may contribute to the detection of several composite events, if it is not consumed – Distinction in event messaging between “multiple receive” and “single receive” – Events which can no longer contribute, e.g. are outdated, should be removed 2. Execution – Actions might have an internal effect i.e. change the knowledge state leading to state transition from (pre)-condition state to post-condition state. – Actions might have an external side effect 54
epts event processing technical society Example: REWERSE MARS • MARS (Modular Active Rules for the Semantic Web) – rule uses specialized heterogeneous event language (E), one or more query languages (Q), a test language (T), and an action language (A) – each of these languages and their constructs are described by metadata and an ontology of its semantics – well-defined interface for communication between the E, Q&T, and A components by variables – SNOOP (E), OWLQ (Q&T) and CCS - Calculus of Communicating Systems (A) 55
epts event processing technical society Example: XChangeEQ • ECA rules ON Event query IF Web query DO Action – the condition part is a declarative query to XML data – the event part is as well a declarative query to XML event data • distributed and web-based: events issued by web nodes (with both emission and reception times) – - action: simple or complex updates send to web nodes • Complex events are expressed as (compound) queries to the event stream – temporal restrictions and event compositions • conjunction, disjunction, exclusion, quantifications, repetitions, and ranks • absolute temporal restrictions, also relative temporal restrictions (within) 56
epts event processing technical society XChangeEQ • The four dimensions of event querying are kept separated: – data extraction (eg oder number, book title) – event composition (eg oder of a dictionary and a grammar both from the same publisher) – temporal (and other) relationships (eg two orders within a week) – event accumulation (eg no default of paiement between the order of the two abovementioned buys) • Equivalent Semantics: – Declarative semantics: Model and fixpoint theories – Operational semantics: CERA (Complex Event Relational Algebra) • XChange^EQ operational semantics: – by language design, the system never wait forever for a complex event to possibly resume. – Incremental evaluation (refining RETE). – Built-in garbage collection of irrelevant complex events (eg events that cannot resume). 57
epts event processing technical society Example: XChange Composite Event andthen [ xchange:event {{ xchange:sender {"http://airline.com"}, cancellation-notification {{ flight {{ number { var Number } }} }} }}, xchange:event {{ xchange:sender {"http://airline.com"}, important {"Accomodation is not granted!"} }} ] within 2 h 58
epts event processing technical society ruleCore Markup Language (rCML) • XML language for ECA rules • Complex Event defined in four sub-elements: – <detect-event>. The <detect-event> element specifies the event that is generated when the composite event is detected. – <no-detect-event>. The <no-detect-event> element specifies the event that is generated when the composite event can not be detected. – <event-selector> The <event-selector> element specifies a logical condition (or filter) for the composite event. – <detector>. The composite event detector consists of a number of event algebra operators (the SNOOP algebra operators) that describe the composite event. 59
epts event processing technical society IBM AMIT Situation Manager Rule Language • Definition of situations and their detection conditions – syntactically equivalent to (complex) event patterns – lifespans which hold references to events with relevant context for a particular pattern calculation • Conceptual event model defines an event type generalization hierarchy – Root event type defines a set of standard attributes e.g., • the event source, creation type, temporal dimensions like occurrence time, detection time and /or transaction time of an event, and quantifications such as status, priority, severity, count – Application specific event types inherit from root event type and define further user-defined attributes – Two types of event relationships • Based-on - a pure syntactical relationship to ease the development process. If an event A is 'basedon' event B, event A inherits the entire structure of event B and can extend it. • Reference - a relationship that allows deriving events without pattern detection. Useful for transformation and enrichment purposes. 60
epts event processing technical society Example AMIT SMRL AMIT SMRL markup language provides an event algebra with semantic notations such as context (lifespan intervals), and semantic association (event group) to define situations <eventTypes> <eventType name="doctorEnterRoom"> <attributeType name="doctor" xsi:type="string"/> <attributeType name="room" xsi:type="integer"/> </eventType> <eventType name="patientEnterRoom"> <attributeType name="patient" xsi:type="string"/> <attributeType name="r_o_o_m" xsi:type="integer"/> </eventType> <eventType name="lockRoom"> <attributeType name="room" xsi:type="integer"/> </eventType> <eventType name="unlockRoom"> <attributeType name="room" xsi:type="integer"/> </eventType> </eventTypes> <lifespan name="roomOpen"> <initiator> <eventInitiator name="unlockRoom"/> </initiator> <terminator> <eventTerminator name="lockRoom"/> </terminator> <keyBy name="sameRoom"/> </lifespan> 61
epts event processing technical society (Reaction) RuleML • General (reaction) rule form that can be specialized as needed • Platform-independent XML-based rule interchange format – translation into platform-specific executable rule languages, e.g. Prova • Three general execution styles: – Active: 'actively' polls/detects occurred events in global ECA style, e.g. by a ping on a service/system or a query on an internal or external event database – Messaging: Waits for incoming complex event message – Reasoning: KR event/action logic reasoning and transitions (as e.g. in Event Calculus, Situation Calculus, TAL formalizations) • Appearance – Global: ‘globally’ defined reaction rule – Local: ‘locally’ defined reaction rule in a specific context 62
epts event processing technical society General Syntax for Reaction Rules <Rule style="active" eval="strong"> <on> <!-- event --> </on> <if> <!-- condition --> </if> <do> <!-- action --> </do> <ifPost> <!-- postcondition --> </ifPost> <doAlternative> <!-- alternative/else action --> </doAlternative> </Rule> 63
epts event processing technical society Selected Reaction RuleML Extended Features • Action Algebra: Succession (Ordered Succession of Actions), Choice (Non-Determenistic Choice), Flow (Parallel Flow), Loop (Loops) • Event Algebra: Sequence (Ordered), Disjunction (Or) , Xor (Mutal Exclusive), Conjunction (And), Concurrent , Not, Any, Aperiodic, Periodic • Support for event / action messaging • Support for different detection, selection and consumption policies • Support for intervals (Time, Event) • Support for situations (States, Fluents) • Support for external event query languages • ... 64
epts event processing technical society 10 minutes break TUTORIAL OUTLINE Introduction Opher Etzion stream processing languages Jon Riecke  rule oriented languages Adrian Paschke Languages Styles Break • Agent oriented languages Opher Etzion • Temporal semantics Jon Riecke Advanced • Integrated development environments Jon Riecke Topics • Formal approaches Adrian Paschke • Conclusion Opher Etzion 65
epts event processing technical society TUTORIAL OUTLINE Introduction Opher Etzion stream processing languages Jon Riecke  rule oriented languages Adrian Paschke Languages Styles  Break • agent oriented languages Opher Etzion • Temporal semantics Jon Riecke Advanced • Integrated development environments Jon Riecke Topics • Formal approaches Adrian Paschke • Conclusion Opher Etzion 66
epts event processing technical society EPA and EPN • In David Luckham’s “the power of events” the term EPN (Event Processing Network) was used as a collection of EPAs (Event Processing Agents). • The EPN describes the c1 p1 admittance Filter: Diabetes p5 c5 EPN Example “programming in the large”, Pattern detection: Alert physician p6 p7 p12 while each individual agent Blood pressure reading p8 Enrich: p10 p11 describes the c2 c6 p2 p9 Is diabetic? p12 Aggregator: Max. “programming in the c7 Fever C p14 Translate: p13 c3 C2F fever p19 Fever F p3 small”. p4 Fever F c8 p15 Pattern Detection: Alert Nurse p16 p20 Pattern Detection: p18 Continuous Fever p17 c9 IBM Haifa Research Lab – Event Processing © 2008 IBM Corporation 67
epts event processing technical society Some EPN, event flow and data flow oriented languages Agent Logic Event Zero Oracle/BEA Spade (IBM System S) IBM InfoSphere Streams Streambase A SPADE program builds a data-flow network out of operators source stream processing sink operator operator operator m str ea ea str m processing stream processing operator operator stre m ea am str source sink operator operator 68 194 Tutorial on the SPADE language © 2008, 2009 IBM Corporation
epts event processing technical society What is the idea ? • The EPN (event processing network) represents the event processing application as a directed graph: – The nodes represent event processing agents (and states in some models) – The edges represent either individual events or event streams (depends on the processing type). A generalized EPN may support both. • The coverage of EPN varies – It may cover some language operator (e.g. SQL like language) – It may take a broader approach and cover also routing decisions, pub/ sub etc… – EPN can have a specific programming model, or an hybrid programming model, where one or more languages represent the agents. 69
epts event processing technical society Anatomy of Agents Performs the processing Selects the events that are Derives output events relevant for the processing detect patterns) (e.g. Selector Processor Deriver 70
epts event processing technical society Event Processing Specialized Agent types Event Processing Agent Filter Transform Detect Pattern Route Translate Aggregate Split Compose Enrich Project 71
epts event processing technical society Routing – channel EPA Itinerary-based routing Subscription-based routing Intelligent routing 72
epts event processing technical society Making the agents context sensitive Time interval that starts by: {event, clock, offset from event} and terminates by: {event, clock, offset from event, count of certain events} – each can have multiple instances (e.g. several events, periodic clock, multiple offsets} When? Examples: [9:00AM, angry-customer-alert], [angry- customer-alert, +1 hour], [request – 2 hours, request] Spatial dimensions/ abstraction Where? Inside building A, In Italy, within 1KM from here (given GPS coordinate) An entity or a collection of entities that determine the partition of the context Who? Class of customer, customer + order, large order… The event happens when some state is in effect Which state? Traffic Jam, Red Alert, Bullish Market 73
epts event processing technical society Filtering Many of the raw events are not relevant for processing, in some applications only a sample is selected for processing. 74
epts event processing technical society Transformation Translation / projection: Translation and simple derivations Aggregation/ composition: statistical aggregator or concatenating events, can be stand alone agent or a scalar derivation. May be in network edge. Splitting: Splits events to multiple events 75
epts event processing technical society Enrichment Enriches the content of events from reference data in databases, spreadsheets, Email messages, text files etc.. 76
epts event processing technical society Pattern Matching Pattern definition example In many cases, the agent does not consume raw events, but event that are derived from pattern matching. DEBS 2008 tutorial on event processing patterns provided a deep dive 77
epts event processing technical society Dimensions - I • Meta-data – EPN and agent definitions may be defined as meta-data – Event schema is implementation dependent • State – State can be represented as another node in the EPN – Some EPAs may be state oriented, e.g. the input event is change in state – State is one of the context dimensions 78
epts event processing technical society Dimensions - II • Execution model: – EPN representation is natural one for concurrency/parallelism since the dependencies are explicit – Since it models explicitly the end-to-end event flow, it serve as a convenient model for QoS related optimizations (latency, throughput, real-time constraints) • Programming model: – Can support multiple programming styles – Can support all types of event processing functionality 79
epts event processing technical society p1 p5 c1 c5 Filter: Diabetes admittance p6 EPN example Pattern detection: p12 Alert physician p7 Blood pressure reading p11 p8 p10 c2 Enrich: p2 Is diabetic? c6 p9 p12 Aggregator: Max. c7 Fever C p14 Translate: p13 c3 C2F fever p19 Fever F p3 Fever F p15 Pattern Detection: p4 Alert Nurse c8 p16 p20 Pattern Detection: p18 Continuous Fever p17 c9 80
epts event processing technical society Event Zero example 81
epts event processing technical society System S Example Congestive Heart Failure Freq alert Arrhythmia Analyzer QRS RR P and T Source PE Glucose alert BP S/D CHF Weight trend IHE detector Adapter *Daby Sow 82
epts event processing technical society Agent Logic example 83
epts event processing technical society TUTORIAL OUTLINE Introduction Opher Etzion stream processing languages Jon Riecke  rule oriented languages Adrian Paschke Languages Styles  Break agent oriented languages Opher Etzion • Temporal semantics Jon Riecke Advanced • Integrated development environments Jon Riecke Topics • Formal approaches Adrian Paschke • Conclusion Opher Etzion 84
epts event processing technical society Varieties of Temporal Semantics • Do events carry timestamps? Where does the timestamp come from (CEP system or external world)? • Granularity? Second, millisecond, microsecond, …? • Point in time or interval of time? • Can events arrive out-of-order from a single data source? • Can events be reordered if they arrive out-of-order? • How long does the system wait? • What happens if system moves ahead, and old event arrives? • In distributed case, is there a difference between emission and reception time? 85
epts event processing technical society Simple Temporal Semantics • Events arrive in order from a single data source • Events have timestamps, but not reordered • Advantage • Process as soon as possible • Disadvantages • Non-deterministic • Hard to manage if time really matters • Emission times de-synchronized in distributed case 86
epts event processing technical society Global Clock • Events have timestamps consistent across data sources • Wait for events for specified time; discard if wait too long • Advantages • Deterministic • Time might be relevant • Disadvantages • Speed of processing • Buffering required for reordering events 87
epts event processing technical society TUTORIAL OUTLINE Introduction Opher Etzion stream processing languages Jon Riecke  rule oriented languages Adrian Paschke Languages Styles  Break agent oriented languages Opher Etzion  Temporal semantics Jon Riecke Advanced • Integrated development environments Jon Riecke Topics • Formal approaches Adrian Paschke • Conclusion Opher Etzion 88
epts event processing technical society Integrated Development Environments (IDEs) • Two main forms • Text based • Visual or graphical based • Not an either-or; most systems have some combination of text and visual display of dataflow 89
epts event processing technical society Visual IDEs • Streams = nodes, flows = edges • Drag-and-drop streams from palette onto canvas and wire together • Configure streams with property sheets or text • Advantages: simple metaphor; easy to see flow • Disadvantages: managing screen real-estate 90
epts event processing technical society Visual IDE Example 91
epts event processing technical society Text-based IDEs • Streams represented in text • Advantages • Appeals to programmers • Can be quicker to write code • Disadvantages • Hard to see the data flow • Hard to find bottlenecks in program 92
epts event processing technical society Text-based IDE 93
epts event processing technical society Another Text-based IDE 94
epts event processing technical society Advanced Features in IDEs • Ability to start/stop programs • Debugging, including viewing intermediate events and setting breakpoints • Tests, including ability to control data feeds • Dashboards for displaying results 95
epts event processing technical society Breakpoints and Debugging 96
epts event processing technical society Performance Monitoring 97
epts event processing technical society Dashboard Construction 98
epts event processing technical society Dashboard Construction 99
epts event processing technical society Assessment: IDEs still have a ways to go • Need more features that programmers have come to expect • Single-stepping with line numbers • Examination of events and data • Forward and backward replay of time • Need to be integrated into more dashboards and other visualization tools 100