EPTS DEBS2011 Event Processing Reference Architecture and Patterns Tutorial v1 2

epts event processing technical society Tutorial v1.2 epts event processing technical society Content by: members of the EPTS Reference Architecture Working GroupEvent Processing Adrian Paschke (Freie Universitaet Berlin) Paul VincentReference Architecture (TIBCO Software) Catherine Moxeyand Design Patterns (IBM) Alex Alves (Oracle)

epts event processing technical societyTutorial • Event Processing is an increasingly “mainstream” view in IT, if not widely in production • Event Processing Functional Reference Architecture (FRA) describes the functions involved in event processing • Functions in the FRA can themselves be described as event processing patterns • Most technology providers can support the FRA and its associated design patterns2

epts event processing technical societyAgenda • Introduction to the EPTS Functional Reference Architecture, and what it contains • Mapping Functions to Patterns: some sample Event Processing Patterns • EPTS member pattern implementation examples • Summary and future work on EP Patterns3

epts event processing technical societyIntroduction to Event Processing • Event-centric view of IT • Events are – Sent and Received – Aggregated, Transformed into Data, Deleted – Processed in queries, rules etc – Cause actions like processes, service invocations, etc4

epts event processing technical societyIntroduction to Event Processing Event Processing, Complex Event Processing, Event Stream Processing Event Processing Rules, etc. Event Type Definitions, Event Abstraction, Event Pattern Detection, Event Composition Event Management etc. etc. Events Derived Events Event Producer Event Consumer (Event Source, (Event sink, event Event Emitter) handler, event listener)Design time Run time Administration

epts event processing technical societyMotivation and Benefits• Motivation – Event Processing is evolving and exploiting many technologies – Potential adopters (stakeholders) need a reference to understand suppliers‟ architectures and solutions.• Benefits – a Reference Architecture predefines customizable abstract frames of reference for specific stakeholder concerns and application domains. • Aids reuse of successful EP architectures for frequently occurring EP design problems – Underlying Architecture is a Reference Model that defines the terminology and components in Event Processing architectures

epts event processing technical societyReference Architecture Viewpoints Viewpoint Viewpoint Element Engineering EP Managing EP Business with EP Architecture Architecture Architecture Concepts How to How to apply? How to utilize / sell / implement? own? Stakeholders Architects / Project Manager Decision Maker, Engineers Customer, Provider Concerns Effective Operational Strategic and construction and Management tactical deployment management Techniques / Modeling, IT (service/appl) Monitoring, Languages Engineering management, Enterprise Decision project Management, management Governance10

epts event processing technical societyStakeholders and Viewpoints covered Viewpoints (for CEP / stateful EP) Stakeholder Component Functional Decision Maker [Not applicable] Inputs, Outputs and / End User Processing Requirements View Architect / Solution Components Functions carried out in Designer View CEP View TBD (2012?) Focus so far11

epts event processing technical societyFunctional Reference Architecture • Architect and Developer perspective – includes the 3 main functions (development, run-time and administration), – targets primarily the automated event processing operations • Run-time functions in 2 main groups: – the event infrastructure (sources and consumers) external to the event processor under consideration, – the event processor.

epts event processing technical societyFunctional View source: definitions of EP Event Processing, Complex Event Processing, Event Stream Processing Event Processing Rules, etc. Event Type Definitions, Event Abstraction, Event Pattern Detection, Event Composition Event Management etc. etc. Events Derived Events Event Producer Event Consumer (Event Source, (Event sink, event Event Emitter) handler, event listener)Design time Run time Administration

epts event processing technical societyReference Architecture: Functional View Definition, Modeling, (continuous) Improvement Event Reaction Assessment, Routing, Prediction, Discovery, Learning (Pattern, Control, Rule, Query, RegEx.etc) Event Process Monitoring, Control 0..* State Management Complex Event Detection Event and Complex Event Consolidation, Composition, Aggregation 0..* Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification 0..* Event Preparation Identification, Selection, Filtering, Monitoring, Enrichment 0..* Event Production Event Consumption Publication, Dashboard, Apps, Retrieval External ReactionDesign time Run time Administration

epts event processing technical societyReference Architecture: Functional View / Runtime Event Reaction Assessment, Routing, Prediction, Discovery, Learning 0..* State Management Complex Event Detection Consolidation, Composition, Aggregation 0..* Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification 0..* Event Preparation Identification, Selection, Filtering, Monitoring, Enrichment 0..* Event Production Event Consumption Publication, Dashboard, Apps, Retrieval External Reaction Run time

epts event processing technical societyReference Architecture: Functional View / RuntimeEvent Production: the source of events for event processing.• Event Publication: As a part of event production, events may be published onto a communication mechanism (eg event bus) for use by event consumers (including participants in event processing). This is analogous to a "push" system for obtaining events.• Event Retrieval: As a part of event production, events may be explicitly retrieved from some detection system. This is analogous to a "pull" system for obtaining events. Event Production Event Consumption Publication, Dashboard, Apps, Retrieval External Reaction

epts event processing technical societyReference Architecture: Functional View / RuntimeEvent Consumption: the process of using events from event publication andprocessing. Event processing itself can be an event consumer, although forthe purposes of the reference architecture, event consumers are meant toindicate downstream consumers of events generated in event processing.• Dashboard: a type of event consumer that displays events as they occur to some user community.• Applications: a type of event consumer if it consumes events for its own processes.• External Reaction: caused through some event consumption, as the result of some hardware or software process. Event Production Event Consumption Publication, Dashboard, Apps, Retrieval External Reaction

epts event processing technical societyReference Architecture: Functional View / RuntimeEvent Preparation: the process of preparing the event and associated payload andmetadata for further stages of event processing.• Entity Identification: incoming events will need to be identified relative to prior events, such as associating events with particular sources or sensors.• Event Selection: particular events may be selected for further analysis. Different parts of event processing may require different selections of events. See also event filtering.• Event Filtering: a stream or list of events may be filtered on some payload or metadata information such that some subset is selected for further processing.• Event Monitoring: particular types of events may be monitored for selection for further processing. This may utilise specific mechanisms external to the event processing such as exploiting event production features.• Event Enrichment: events may be "enriched" through knowledge gained through previous events or data. Event Preparation Identification, Selection, Filtering, Monitoring, Enrichment

epts event processing technical societyReference Architecture: Functional View / Runtime Event Analysis: the process of analysing suitably prepared events and their payloads and metadata for useful information. • Event Analytics: the use of statistical methods to derive additional information about an event or set of events. • Event Transforms: processes carried out on event payloads or data, either related to event preparation, analysis or processing. • Event Tracking: where events related to some entity are used to identify state changes in that entity. • Event Scoring: the process by which events are ranked using a score, usually as a part of a statistical analysis of a set of events. See also Event Analytics • Event Rating: where events are compared to others to associate some importance or other, possibly relative, measurement to the event. • Event Classification: where events are associated with some classification scheme for use in downstream processing. Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification

epts event processing technical societyReference Architecture: Functional View / RuntimeComplex Event Detection: the process by which event analysis results in thecreation of new event information, or the update of existing complex events.• Event Consolidation: combining disparate events together into a "main" or "primary" event. See also event aggregation.• Event Composition: composing new, complex events from existing, possibly source, events.• Event Aggregation: combining events to provide new or useful information, such as trend information and event statistics. Similar to event consolidation. Complex Event Detection Consolidation, Composition, Aggregation

epts event processing technical societyReference Architecture: Functional View / RuntimeEvent Reaction: the process subsequent to event analysis and complex eventdetection to handle the results of analysis and detection.• Event Assessment: the process by which an event is assessed for inclusion in some process, incorporation in some other event, etc.• Event Routing: the process by which an event is redirected to some process, computation element, or other event sink.• Event Prediction: where the reaction to some event processing is that some new event is predicted to occur.• Event Discovery: where the reaction to some event processing is the disclosure of a new, typically complex, event type. • Note that event prediction is predicting some future event, usually of a known type, whereas event discovery is the uncovering of a new event type. See also event-based learning.• Event-based Learning: the reaction to some event processing that uses new event information to add to some, typically statistical-based, understanding of events. • Note that event-based learning is a specialisation of general machine learning and predictive analytics. Event Reaction Assessment, Routing, Prediction, Discovery, Learning

epts event processing technical societyReference Architecture: Functional View / Design time Definition, Modeling, (continuous) Improvement Covers the definition, modeling, improvement / (Pattern, Control, Rule, Query, RegEx.etc) maintenance of the artifacts used in event processing: Event and Complex Event • event definitions, including event metadata and payloads, • event and event object organisations and structures, • event processing transformations / queries / rules / procedures / flows / states / decisions / expressions (although these can sometimes be considered as administrative updates in some situations)Design time

epts event processing technical societyReference Architecture: Functional View / Administration Administrative concepts of Event Process Monitoring, Control monitoring and control. This may involve • starting and stopping the application and event processing elements, including application monitors • providing and updating security levels to event inputs and outputs (also can design-time) • management of high availability and reliability resources, such as hot standby processes • resource utilisation monitoring of the event processing components • process updates, such as how-swapping of event processing definitions to newer versions. Administration

epts event processing technical societyEvent Processing Patterns • Functions from Reference Architecture are a guide to possible low-level patterns Event Reaction Assessment, Routing, Prediction, Discovery, Learning Complex Event Detection Consolidation, Composition, Aggregation Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification Event Preparation Identification, Selection, Filtering, Monitoring, Enrichment25

epts event processing technical societyWhich Classification of Patterns to Use? • Common EP Functions? As described in the Reference Architecture • Other sources? Recent literature26

epts event processing technical societyComparison of EPTS RA Fns and EPIA EP Agents Event Reaction Assessment, Routing, Prediction, Discovery, Learning Complex Event Detection Consolidation, Composition, Aggregation Event Analysis Analytics, Transforms, Tracking, EPA Scoring, Rating, Classification Event Preparation Filter Transformation Pattern detectIdentification, Selection, Filtering, Monitoring, Enrichment Translate Aggregate Split Compose Enrich Project27

epts event processing technical societyComparison of EPTS RA Fns and EPIA EP AgentsEvent Preparation28

epts event processing technical societyComparison of EPTS RA Fns and EPIA EP AgentsEvent Analysis29

epts event processing technical societyComparison of EPTS RA Fns and EPIA EP AgentsComplex Event Detection30

epts event processing technical societyComparison of EPTS RA Fns and EPIA EP AgentsEvent Reaction31

epts event processing technical societyWhat do we want to cover in an EP Pattern? Type / Operands Operation / synonym Operators Event Filtering: a stream or list of events may be filtered on some payload or metadata information such that some subset is selected for further processing. Rationale / Mapping to business rule EPL template32

epts event processing technical societyEP MetaPattern version 1 • Name & Alternative Names • Classification versus other references – EPTS RA and other references e.g. “EP in Action” book • Description – Role, and Business Rule type specification / requirement • Structure – EPTS Glossary terms • Implementations33

epts event processing technical societyEP MetaPattern future versions? • Name & Alternative Names • Classification versus other references – EPTS RA and other references e.g. “EP in Action” book • Description may use revised EP Fns classification / ontology – Role, and Business Rule type specification / requirement • Structure – EPTS Glossary terms • Implementations may use EPL classification per EPTS Languages WG34

epts event processing technical societyEP Pattern Implementations • Differ in Implementation type (Event Processing Language, Execution semantics, etc) • Sampled systems we cover: 1.TIBCO BusinessEvents Rete-production-rule-engine (with continuous query and event-expression options) 2.Oracle CEP Stream-processing engine 3.Prova Logic Programming Semantic CEP Rule Engine 4.IBM WebSphere Decision Server event-driven production-rule-engine & IBM Infosphere Streams stream processing engine35

epts event processing technical societySome EP Patterns to review in more detail 1. Filter: filter out all events that have some property in their payload / data – E.g. customer purchases: filter out those with values < $100 2. Enrichment: add some information based on prior events – E.g. customer purchase: add the customer history to the purchase event 3. Aggregation: add prior events together for new information – E.g. customer purchase history: track the sum total of order values for a customer 4. Routing: based on the purchase type pass the event on to the appropriate service – E.g. customer purchase event: pass on to a provisioning service37 based on the type of product / product classification

epts event processing technical societySome EP Patterns to review in more detail 1. Filter: filter out all events that have some property in their payload / data – E.g. customer purchases: filter out those with values < $100 2. Enrichment: add some information based on prior events – E.g. customer purchase: add the customer history to the purchase event 3. Aggregation: add some information based on prior events – E.g. customer purchase history: track the sum total of order values for a customer 4. Routing: based on the purchase type pass the event on to the appropriate service – E.g. customer purchase event: pass on to a provisioning38 service based on the type of product / product classification

epts event processing technical societyFilter Pattern: Classification (1) • EPTS Reference Architecture: – "During event preparation, a stream or list of events may be filtered on some payload or metadata information such that some subset is selected for further processing." • EPIA: – Filter (EPA or Event Processing Agent) performs filtering only and has no matching or derivation steps, so it does not transform the input event.39

epts event processing technical societyFilter Pattern: Description (2) • Role / Explanation – Comparing some property of the event (an attribute or metadata) with some other value (from some other event, or data) • Associated Business Rule Specification – As a constraint: a selection (to be enforced during the processing of events): All <event entities> that have Filter <filter expression> event event f(event) must have <some status>. discarded event40

epts event processing technical societyFilter Pattern: Structure (3) • EPTS Glossary comparison – A filter pattern can relate to several terms in the EPTS Glossary • can be specified in an event pattern (A template containing event templates, relational operators and variables...) • .. by an event pattern constraint (A Boolean condition that must be satisfied by the events observed in a system...) • .. as part of an event processing rule (A prescribed method for processing events.) • .. or as part of event stream processing (Computing on inputs that are event streams.) –41

epts event processing technical societyFilter Pattern: Implementations (4) • TIBCO, Oracle, Prova and IBM implementations for this pattern Filter event event f(event) discarded event42

epts event processing technical societyFilter Pattern: Implementation 1aTIBCO BusinessEvents: rules • General pattern: <declare> event event <if> event-boolean-expression Filter f(event) <then> update-event-with-fact event OR action-for-filtered-event Operation on • Effect: ignore filtered events event on operations on non-filtered events43

epts event processing technical societyFilter Pattern: Implementation 1bTIBCO BusinessEvents: event removal rules • Implementation type: Rete-based rule-driven system • Alternate pattern: <declare> event event <if> NOT Filter NOT event-boolean-expression !f(event) <then> consume-event event • Effect: removed filtered events from system (and ALL subsequent operations!)44

epts event processing technical societyFilter Pattern: Implementation 1cTIBCO BusinessEvents: State transitions • General pattern: <state models for concept C> event <route from State n to n+1> <on> event Filter <where> event-filter query(event) • Effect: change state for some concept State transition • Notes: this is a specialisation of the general rule pattern!45

epts event processing technical societyFilter Pattern: Implementation 1dTIBCO BusinessEvents: Queries • General pattern: <select> output event <from> event <where> event-filter Filter query(event) <policy> notification Policy Event event subset • Effect: collect filtered events for event operations per some policy Operations on event subset • Note: the pattern matching framework on policy event has a similar notion46 but a different surface syntax

epts event processing technical societyFilter Pattern: Implementation 2aOracle CEP: queries • General pattern: event SELECT <property> FROM <STREAM> predicate WHERE <predicate-condition> event • Effect: discard events whose predicate Operation on evaluate to false event47

epts event processing technical societyFilter Pattern: Implementation 2bOracle CEP: model • Scenario: – Select only stocks from the stock tick stream whose symbols are „AAA‟. • Application Model48

epts event processing technical societyFilter Pattern: Implementation 2cOracle CEP: query Specify STREAM source SELECT * FROM StockTickStream Define predicate for WHERE symbol = „AAA‟ filtering49

epts event processing technical societyFilter Pattern: Implementation 3aProva: rules• Implementation type: backward reasoning logical filter• General pattern template: rcvMsg / rcvMult <Event Msg Pattern(s)> :- event <logical filter condition(s)>, … event ... . Event pattern + <logical derivation rule 1> :- logical rule filter <filter conditions>. Backward <logical derivation rule 2> :- Resoning Rule event <filter conditions>. chaning ...• Effect: Unify received events with Operation on event pattern definition and apply logical event50 filter rules (derivation rules with rule chaining)

epts event processing technical societyFilter Pattern: Implementation 3bProva: Messaging Reaction Rules in Prova• Send a messagesendMsg(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)• Receive a messagercvMsg(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)• Receive multiple messagesrcvMult(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)Description: – XID is the conversation identifier – Protocol: protocol e.g. self, jms, esb etc. – Agent: denotes the target or sender of the message – Performative: pragmatic context, e.g. FIPA Agent Communication – [Predicate|Args] or Predicate(Arg1,..,Argn): Message payload

epts event processing technical societyFilter Pattern: Implementation 3cProva: Example % Filter for stocks starting with „A“ and price > 100 rcvMult(SID,stream,“S&P500“, inform, tick(S,P,T)) :- S = "A.*", P > 100, sendMsg(SID2,esb,"epa1", inform, happens(tick(S,P),T).Example with rule chaining % Filter for stocks starting with „A“ and price > 100 rcvMult(SID,stream,“S&P500“, inform, tick(S,P,T)) :- filter(S,P), sendMsg(SID2,esb,"epa1", inform, happens(tick(S,P),T). filter(Symbol,Price) :- Price > 100, Symbol = "A.*".

epts event processing technical society Filter Pattern: Implementation 3d Prova: Semantic Event Processing Example Semantic Query Filer: Stocks of companies, which have production facilities in Europe and produce products out of metal and Have more than 10,000 employees. Event Stream {(Name, “OPEL”)(Price, 45)(Volume, 2000)(Time, 1) } {(Name, “SAP”)(Price, 65)(Volume, 1000) (Time, 2)} {(OPEL, is_a, car_manufacturer), (car_manufacturer, build, Cars), (Cars, are_build_from, Metall), Semantic (OPEL, hat_production_facilities_in, Germany),Knowledge Base (Germany, is_in, Europe) (OPEL, is_a, Major_corporation), (Major_corporation, have, over_10,000_employees)} rcvMult(SID,stream,“S&P500“, inform, tick(Name^^car:Major_corporation,P^^currency:Dollar, T^^time:Timepoint)) :- … <semantic filter inference> .

epts event processing technical society Filter Pattern: Implementation 4a IBM WebSphere Decision Server: Business Events component event event • Example: Define a filter to only process events with a Filter purchase value greater than £100 event Operation on • Effect: Only process events event which match the filter55

epts event processing technical societyFilter Pattern: Implementation 4bIBM InfoSphere Streams: Streams Processing Language • Example: composite Main { graph stream<rstring name, uint32 age> Beat = Beacon() {} stream<rstring name, uint32 age> Youngs = Filter(Beat) { param filter : age < 30u; } (stream<Beat> Younger; stream<Beat> Older) = Filter(Beat) { param filter : age < 30u; } } • Effect: filter out into streams of older and younger ages 56

epts event processing technical societyEnrichment Pattern: Classification (1) • Alternative Names: – Event Annotation, Semantic Event • EPTS Reference Architecture: – "During event preparation, events may be "enriched" through knowledge gained through previous events or data. " • EPIA: – Enrich (EPA): a translate EPA that takes a single input event, uses it to query data from a global state element, and creates a derived event which includes the attributes from the original event, possibly with modified values...58

epts event processing technical societyEnrichment Pattern: Description (2) • Role / Explanation – Updating some attribute or metadata of the event with some other values, possibly computed, from some other event or data • Associated Business Rule Specification – a selection (to be enforced during the processing of events): All <event entities> that are also <enriched data expression> Other data must have <some status>. or event Enrichment event event f(event)59

epts event processing technical societyEnrichment Pattern: Structure (3) • EPTS Glossary comparison – Enrichment can be specified in an event pattern (pattern definition containing additional meta data, semantic knowledge) – .. as part of an event processing rule (A method for enriching events during processing, e.g., by analysing the meta data and querying external data sources or inferencing semantic knowledge bases) – .. or as part of event stream processing (additional additional information to event stream data, e.g. by querying external data sources)60

epts event processing technical societyEnrichment Pattern: Implementations (4) • TIBCO, Oracle, Prova and IBM implementations for this pattern Other data or event Enrichment event event f(event)61

epts event processing technical societyEnrichment Pattern: Implementation 1TIBCO BusinessEvents: Rules • General pattern: Enrich <declare> event, data data event <if> event-data-match Identify enrichment data <then> update-event-with-data Enrich Operation • Effect: find appropriate data for on event events and apply enrichment operations on them Event62

epts event processing technical societyEnrichment Pattern: Implementation 2aOracle CEP: queries • General pattern template: event SELECT <property> FROM <STREAM>[WIN],<RELATION> join-condition WHERE <join-condition> event • Effect: enrich events from stream with contextual data from relation where condition applies. Operation on event63

epts event processing technical societyEnrichment Pattern: Implementation 2bOracle CEP: model • Scenario: – For every stock event from stock tick stream whose bid is greater than 5.0, find its company‟s location. • Application Model64

epts event processing technical societyEnrichment Pattern: Implementation 2cOracle CEP: queries No need for window SELECT event.symbol, location operator on a „relation‟ FROM StockTickStream [NOW] AS event, StockTable AS data WHERE event.symbol = data.symbol AND event.lastBid > 5.065

epts event processing technical societyEnrichment Pattern: Implementation 2dOracle CEP: table metaphor StockTable Symbol Full-name Location AAA The AAA company San Francisco BBB The BBB company San Jose Time Input (StockTickStream) Output (QUERY) 1 {“AAA”, 10.0, 10.5} {“AAA”, “San…”} 2 {“BBB”, 11.0, 12.5} {“BBB”, “San Jose”} 3 {“AAA”, 4.0, 4.5}66

epts event processing technical societyEnrichment Pattern: Implementation 3aProva: rules• General pattern template: rcvMsg / rcvMult <Event Msg Pattern(s)> :- event <query built-in(s)>, <enrichment condition(s)> External ... . Query data <logical enrichment rule 1> :- <enrichment operations>. Enrichment rules• Effect: query data from external source event* and enrich the event Operation on enriched event67

epts event processing technical societyEnrichment Pattern: Implementation 3bProva: External Data and Object Integration• Java ..., L=java.util.ArrayList(),, ..., java.lang.String(“Hello”)• File Input / Output ..., fopen(File,Reader), ...• XML (DOM) document(DomTree,DocumentReader) :- XML(DocumenReader),• SQL …,sql_select(DB,cla,[pdb_id,“1alx“],[px,Domain]).• RDF ...,rdf(http://...,"rdfs",Subject,"rdf_type","gene1_Gene"),• XQuery ..., XQuery = for $name in StatisticsURL//Author[0]/@name/text() return $name, xquery_select(XQuery,name(ExpertName)),• SPARQL ...,sparql_select(SparqlQuery,name(Name),class(Class), definition(Def)),

epts event processing technical societyEnrichment Pattern: Implementation 3cProva: Example with SPARQL Query% Filter for car manufacturer stocks and enrich the stock tick event with data from Wikipedia (DBPedia) about the manufacturer and the luxury carsrcvMult(SID,stream,“S&P500“, inform, tick(S,P,T)) :- carManufacturer(S,Man), % filter car manufacturers findall([luxuryCar|Data],luxuryCar(Man,Name,Car),Data), % query EnrichedData = [S,Data], % enrich with additional data sendMsg(SID2,esb,"epa1", inform, happens(tick(EnrichedData,P),T).% rule implementing the query on DBPedia using SPARQL queryluxuryCar(Manufacturer,Name,Car) :- Query="SELECT ?manufacturer ?name ?car % SPARQL RDF Query WHERE {?car <http://purl.org/dc/terms/subject> <http://dbpedia.org/resource/Category:Luxury_vehicles> . ?car foaf:name ?name . ?car dbo:manufacturer ?man . ?man foaf:name ?manufacturer. } ORDER by ?manufacturer ?name“,sparql_select(Query,manufacturer(Manufacturer),name(Name),car(Car)).

epts event processing technical society Enrichment Pattern: Implementation 4 IBM WebSphere Decision Server: Business Events component • Example: Define a Mapped Key expression, where a field in the event is used to look up objects in a database table event Mapped Key look up Enriched event • Effect: required enrichment data is obtained70 from the data base and added to event

epts event processing technical societyAggregation Pattern: Classification (1) • Alternative Names: – Event Summarization, Composite event, Complex event • EPTS Reference Architecture: – "During complex event detection, combining events to provide new or useful information, such as trend information and event statistics. Similar to event consolidation " • EPIA: – Aggregate (EPA): a transformation EPA that takes as input a collection of events and creates a single derived event by applying a function over the input events.72

epts event processing technical societyAggregation Pattern: Description (2) • Role / Explanation – Aggregation is used to describe several events as a single composite event, generally summarizing a metric of a set of component events. – For example: generate a (composite) event that contains the medium price of a stock over a 10 minute stream of events. • Associated Business Rule Specification – a summarisation (to be enforced during event processing): The <aggregation fn> of <event entities> that are <selection constraint> must have Aggregation <some constraint>. event event event f(events)73 event

epts event processing technical societyAggregation Pattern: Structure (3) • EPTS Glossary comparison – The term aggregate event is sometimes used for some forms of composite or derived event. – Composite event: a derived, complex event • is created by combining base events using a specific set of event constructors such as disjunction, conjunction, sequence, etc. • always includes the base (member) events from which it is derived. – Derived event (/synthesized event): an event that is generated as a result of applying a method or process to one or more other events.74

epts event processing technical societyAggregation Pattern: Implementations (4) • TIBCO, Oracle, Prova and IBM implementations for this pattern Aggregation event event event f(events) event75

epts event processing technical societyAggregation Pattern: Implementation 1TIBCO BusinessEvents: Rules • General pattern: Aggregation <declare> event, object event aggregation-object Identify <if> event for event-member-of-aggregation aggregation <then> update-aggregation-object Aggregate with event-data Operation on event • Effect: construct aggregation Updated (event) object for each incoming event Aggregation76 Object

epts event processing technical societyAggregation Pattern: Implementation 2aOracle CEP: queries • General pattern template: Aggregation object event SELECT <summary-property> FROM <STREAM>[WIN] Identify WHERE <predicate> event for GROUP BY <aggregation-identity> aggregation • Effect: summarizes the properties of Aggregate several simple event into a new Operation complex event on event Updated Aggregation77 Object

epts event processing technical societyAggregation Pattern: Implementation 2bOracle CEP: model • Scenario: – Output the average bid and ask price of a stock in the last 10 seconds. • Application Model78

epts event processing technical societyAggregation Pattern: Implementation 2cOracle CEP: queries SELECT symbol, AVG(bid), AVG(ask) FROM StockTickStream [RANGE 10 SECONDS] GROUP BY symbol79

epts event processing technical societyAggregation Pattern: Implementation 2dOracle CEP: table metaphor Time Input Output 1s {“AAA”, 10.0, 12.0} {“AAA”, 10.0, 12.0} 4s {“AAA”, 12.0, 14.0} {“AAA”, 11.0, 13.0} {“AAA”, 11.0, 13}, 9s {“BBB”, 4.0, 5.0} {“BBB”, 4.0, 5.0} 15s {“BBB”, 8.0, 10.0} {“BBB”, 6.0, 7.5} > 10s 20s {“BBB”, 8.0, 10.0}80

epts event processing technical society Aggregation Pattern: Implementation 3a Prova: rules Aggregation object event• General pattern template: <rule_head> :- Check & Update <Create Aggregator>, Timer / Size Count @group(<reaction group>) @timer|size(Start,Interval,Aggregator) Identify event rcvMsg <EventPattern> [<Aggregate Operation>]. aggregation for or <rule_head> :- Aggregate @or(<reaction group>) Operation rcvMsg <Aggregator>, on event ... <consume Aggregator>.• Effect: Repeated incremental Updated Aggregation Consume Aggregation aggregations over new events Object Object 81

epts event processing technical societyAggregation Pattern: Implementation 3bProva: Example with Time Counter % This reaction operates indefinitely. When the timer elapses (after 25 ms), the groupby map Counter is sent as part of the aggregation event and consumed in or group, and the timer is reset back to the second argument of @timer.groupby_rate() :- Counter = ws.prova.eventing.MapCounter(), % Aggr. Obj. @group(g1) @timer(25,25,Counter) % timer every 25 ms rcvMsg(XID,stream,From,inform,tick(S,P,T)) % event [IM=T,Counter.incrementAt(IM)]. % aggr. operationgroupby_rate() :- % receive the aggregation counter in the or reaction @or(g1) rcvMsg(XID,self,From,or,[Counter]), ... <consume the Counter aggreation object>.

epts event processing technical societyAggregation Pattern: Implementation 4aIBM WebSphere Decision Server: Business Eventscomponent• General Pattern: event event Define how the events should be accumulated Add to array ... then when processing the events, use the Intermediate relevant Object Array aggregator• Effect of the example: sum up the value of a Operation customer‟s purchases over a week on aggregate 83

epts event processing technical societyAggregation Pattern: Implementation 4bIBM InfoSphere Streams: Streams Processing Language • Example: // sliding window stream<Beat, tuple<int32 maxSalary, uint32 ageOfMaxSalary> > Agg4 = Aggregate(Beat) { window Beat : sliding, time(10.5), count(10); output Agg4 : maxSalary = Max(salary), ageOfMaxSalary = ArgMax(salary, age); } • Effect of example: Aggregate the salary values in the stream within a time-based window. 84

epts event processing technical societyRouting Pattern: Classification (1) • Alternative Names: – Event Summarization, Composite event, Complex event • EPTS Reference Architecture: – "During event reaction, event routing is the process by which an event is redirected to some process, computation element, or other event sink. " • EPIA: no direct analogy, possibly maps to split / compose / project – Project (EPA): a translate EPA that takes an input event and creates a single derived event containing a subset of the attributes of the input event.86

epts event processing technical societyRouting Pattern: Description (2) • Role / Explanation – Routing is used to describe the process of adding one or more new destinations to an event. – For example: route an input event to the appropriate specialist agent for that event type. • Associated Business Rule Specification – a constraint (to be enforced during event processing): The <event> that Other data satisfies <selection constraint> or event must be assigned to <destination>. Routing event event dest = f(event+data)87

epts event processing technical societyRouting Pattern: Structure (3) • EPTS Glossary comparison – Routing (a process on events) is not defined, but is related to associating an event to an event sink. – Event sink (event consumer) is an entity that receives events. • Examples: • ƒ Software module • ƒ Database • ƒ Dashboard • ƒ Person88

epts event processing technical societyRouting Pattern: Implementations (4) • TIBCO, Oracle, Prova and IBM implementations for this pattern Other data or event Routing event event dest = f(event+data)89

epts event processing technical societyRouting Pattern: Implementation 1TIBCO BusinessEvents: Rules • General pattern: Routing <declare> event, data event routing-data Identify <if> event for event-matches-routing-data routing <then> Create routed create-new-routed-event and event send-new-routed-event event • Effect: match events to routing rule and construct / send new event90

epts event processing technical societyRouting Pattern: Implementation 2aOracle CEP: query pattern • General pattern template: Routing data event Route 1: <filtering pattern> query Route 2: <filtering pattern> query Identify Route n: <filtering pattern> query event for • Effect: Different predicates routing associated to different queries evaluate and select appropriate Select destination destinations event91

epts event processing technical societyRouting Pattern: Implementation 2bOracle CEP: model • Scenario: – Output stocks whose symbols start with „a-m‟ to destination 1, and whose symbols start with „n-z‟ to destination 2. • Application Model92

epts event processing technical societyRouting Pattern: Implementation 2cOracle CEP: queries Destination-1: SELECT * FROM StockTickStream WHERE symbol.matches(“^[a-m]”) Destination-2: SELECT * FROM StockTickStream WHERE symbol.matches(“^[n-z]”)93

epts event processing technical societyRouting Pattern: Implementation 3 eventProva: rules • General pattern template: Identify event for rcvMsg <Event Msg Pattern> :- routing <routing decisions>, sendMsg <route event >. Routing Routing Decisions data <routing decision rule 1> :- <decision conditions>. Create routed <routing decision rule 2> :- event <decision conditions>. ... Route event event • Effect: Events are routed according to the routing decision94 rules

epts event processing technical societyRouting Pattern: Implementation 3bProva: Example with Agent (Sub-) ConversationsrcvMsg(XID,esb,From,query-ref,buy(Product) :- routeTo(Agent,Product), % derive processing agent % send order to Agent in new subconversation SID2 sendMsg(SID2,esb,Agent,query-ref,order(From, Product)), % receive confirmation from Agent for Product order rcvMsg(SID2,esb,Agent,inform-ref,oder(From, Product)).% route to event processing agent 1 if Product is luxuryrouteTo(epa1,Product) :- luxury(Product).% route to epa 2 if Product is regularrouteTo(epa2,Product) :- regular(Product).% a Product is luxury if the Product has a value over …luxury(Product) :- price(Product,Value), Value >= 10000.% a Product is regular if the Product ha a value below

epts event processing technical societyRouting Pattern: Implementation 4IBM: Business Events component of WebSphereDecision Server Routing event• Example: Define an interaction set, information which will send the request to supplier A, when the purchase indicates supply from Supplier A Identify• event routing event• Effect of example: Events are routed as required Consume 96 event

eptsevent processing technical society Another Pattern Type Other patterns may be required for implementation, such as handling produced-consumer interactions

epts event processing technical societyApplication-Time Pattern • Problem: – Need to use application‟s view of time, instead of CPU wall-clock. – This is particularly useful when events originate from different machines and need some way of synchronizing. • Scenario: – Again, consider bid and ask stream… – However, bid and ask requests are time-stamped on trader at the time that the order is placed.99

epts event processing technical societyApplication-Time Pattern • Scenario: – Seller places two ask requests respectively at time 8:00:00 and 8:00:12 – Buyer places one bid request at time 8:00:11100

epts event processing technical societyApplication-Time Pattern • Scenario: – Remember that we want to correlate using a 10 seconds window, as anything older is stale… – Hence first event from seller should not be considered, and we should correlate the ask price of 11.0 with the bid price of 9.5101

epts event processing technical societyApplication-Time Pattern • Scenario: – However, the reality is that the first two events could arrive together in a burst in the exchange, and the third could be delayed in the cloud…102

epts event processing technical societyApplication-Time Pattern • Scenario: – In this case, bid price of 9.5 would correlate to ask price of 10.0 and the exchange would loose money as the spread is lower…103

epts event processing technical societyApplication Time-stamped Pattern: Implementation 1Oracle CEP • Solution: – The query does not change… – However STREAMS must be configured to use application time-stamps based upon some event property, instead of having events being system- timestamped…104

epts event processing technical societyApplication Time-stamped Pattern: Implementation 2IBM Business Monitor• Example: Define a correlation expression „Order Placed‟ „Order Complete‟ event event• Effect of example: The duration of a customer order can be obtained from the timestamps on start and end events Correlate Calculate Duration 105

epts event processing technical societyApplication Time Pattern: Implementation 3Prova: rules• General pattern template: EventrcvMsg <Event Msg Pattern(s)> :- <add application timestamp to event>, sendMsg <Adjusted Event>. adjust event• Effect: Any timestamp can be adjusted (for time computations) Updated Event106

epts event processing technical societyApplication Time-stamped Pattern: Implementation 4TIBCO BusinessEvents: PreProcessor Pattern• General Pattern: adjust Event metadata (e.g. Timestamp) in preprocessor for that event Event – events are generally non-monotonic! – PreProcessor applies regardless of engine used! Preprocess:• Effect: adjust event Any custom timestamp can be adjusted (for time computations) Note: If the problem is “out of order” events Updated Event (rather than synchronizing clocks) a different pattern may be better! 107

eptsevent processing technical society Use Cases Illustrating using some of the patterns

epts event processing technical societyIBM Use Case: Utilizing Smart MetersLarge energy company improved customer responsivenessusing event processing Use Case • Utilized events (and lack of events) from smart meters to detect service outages and disruptions • Empowered operations team to define outage detection and response patterns without coding or scripting Smarter Business Outcomes • Automated detection and response to service Challenge: Reliance on customer calls outages to detect and respond to outages and reduce time to recovery while providing • Improved customer satisfaction by delivery of more information to the customer estimated time to recovery alerts based on customer preferences

epts event processing technical societyEnergy Company Leveraging Smart Meters to ImproveCustomer Responsiveness Customer Business Data Events 2 1 3 Data Collection Engine (DCE) WebSphere Business Events 1 4 Tivoli OMNIbus, Network Manager, and Impact Power Outage Disruption Management Customer Care System (OMS) Manager(s) with IBM Business Monitor Location, Estimated Duration of Disruption Alert and Assist Impacted Customers 110

epts event processing technical societyOracle Use-case: Capital Markets • Low-Latency Trading – Algorithm Trading • Trade Matching – High throughput – In-flight fraud • Position Capture and Aggregation • Risk and Analytics – Pre/post trade analytics – On-demand risk management111

epts event processing technical societyTIBCO Use Case: Application Service GatewayTelco high performance service policies usingevent processing Excellent service qualityMobile Services needed to avoiddifficult to monitor customer churn end to end Auto detect and fixing of issues Service quality affects for better take-up of new services112 user service

epts event processing technical societyService Gateway architecture exploits event processing • Standardized channel for monitoring subscribers • Standardized access to services (e.g. Facebook, Google, etc) • Consolidation of existing multiple gateways • Allows analysis of service usage, subscriber behavior, service performance, marketing campaign success, etc • Creates events for revenue generation and reconciliation • Handles Common Telco Notifications - change-of-device, contract, service- allowing UpSell/Cross sell etc

epts event processing technical societyService Gateway architecture exploits event processing • Standardized channel for monitoring subscribers • Standardized access to services (e.g. Facebook, Google, etc) Security Pattern • Consolidation of existing multiple gateways Decision / Policy • Allows Pattern analysis of service usage, subscriber behavior, service performance, marketing campaign success, etc Routing Pattern • Creates events for revenue generation and reconciliation Transformation • Handles Common Telco Patterns Notifications - change-of-device, contract, service- allowing UpSell/Cross sell etc

epts event processing technical societySummary: Realization Patterns / Idioms Event Reaction Assessment, Routing, Prediction, Discovery, Learning State Management 0..* Complex Event Detection Consolidation, Composition, Aggregation 0..* Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification 0..* Event Preparation Identification, Selection, Filtering, Monitoring, Enrichment 0..* Event Production Event Consumption Application Time Dashboard, Apps, Publication, Retrieval External Reaction

epts event processing technical societyCategorization of Event Processing Patterns Source:Paschke, A.: Design Patterns for Complex Event Processing, DEBS08, Rome, Italy, 2008 http://arxiv.org/abs/0806.1100v1

epts event processing technical society Categorization of Patterns• Categorization according to Good and Bad Solutions – CEP Patterns – CEP Anti-Patterns• Categorization according to the Abstraction Level – Guidelines and Best Practices – Management patterns – Architecture patterns – Design patterns – Mapping patterns – Idioms / Realization patterns – Smells / Refactoring patterns• Categorization according to the Intended Goal – Adoption patterns – Business patterns – Integration patterns – Composite patterns: – …• Categorization according to the Management Level – Strategic patterns – Tactical patterns Source:Paschke, A.: A Semantic Design Pattern Language for Complex Event Processing, – Operational patterns Intelligent Event Processing, Papers from the 2009 AAAI Spring Symposium (2009) , p. 54-60. http://www.aaai.org/Papers/Symposia/Spring/2009/SS-09-05/SS09-05-010.pdf

epts event processing technical societyNext Steps• Continue evolution of EP Reference Architecture – Eg Logical Architecture vs Systems Description• Evolve the EP Pattern <<metapattern>> to be more useful for developers and tools• Expand coverage to all Functions as Patterns, + extend Functions as necessary• Planned Outputs: – EPTS-RA Description white paper – Wiley Book "Pattern Oriented Software Architecture: Architectures, Models and Patterns for Event Processing" (already in preparation) – Input for Event Processing Metamodels and associated standards

epts event processing technical societySummary • Reference Architecture provides a common set of EP Functions that may be mapped to other EP ontologies / taxonomies • Provides the basis for discussing EP Patterns • EP Patterns can be semantic, design or runtime focused • Further evolution of EP Pattern and examples is likely!120

epts event processing technical society Thank you ! Acknowledgment to the Event Processing Technical Society Reference Architecture working group members121

epts event processing technical societyReferences• EPTS Glossary http://www.ep-ts.com/component/option,com_docman/task,doc_download/gid,66/Itemid,84/

EPTS DEBS2011 Event Processing Reference Architecture and Patterns Tutorial v1 2

by isvana

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 5

Statistics

EPTS DEBS2011 Event Processing Reference Architecture and Patterns Tutorial v1 2 Presentation Transcript

http://a0.twimg.com	4
http://paper.li	1