The document provides an overview of event processing, including its history, current state, and future trends. It discusses event processing concepts like events, patterns, context, and languages. It also outlines some challenges like ordering events in distributed systems and dealing with uncertainty. The future trends highlighted include expanding applications of event processing, diversifying platforms and quality of service needs, standardizing aspects of event processing, and making it more accessible to non-programmers.

1. Event processing – past, present, future VLDB 2010 Tutorial, Singapore, September 15 th , 2010 Opher Etzion (opher@il.ibm.com)

2. An example to kick-off this tutorial: Act: Passenger has been re-routed to another destination – send the luggage Act: Bag has reached to the wrong aircraft Notify: Bag has been checked but did not reach the ULD within 20 minutes Notify: Bag has been checked but did not reach the connecting flight THE LUGGAGE PERSPECTIVE: Across the 24 largest airlines more than 5.6 million bags went missing in 2006, this is an average of 15.7 bags per 1,000 travelers. 15% of the bags are never found. BBC News, April 4, 2007 Event processing can help here..

3. Outline of this tutorial What is behind event processing ? How is it related to other computing terms? Where are its roots? Event processing – architecture, building blocks The present: State of the practice in event processing – languages, implementation issues, challenges in implementing event processing applications The future: Trends and research challenges

4. What is behind event processing ? How is it related to other computing terms? Where are its roots?

5. What is “event processing” anyway? or Event processing is a form of computing that performs operations on events

6. Event processing and Data stream management? Aliases? One of them subset of the other? Totally unrelated concepts?

7. In computing we processed events since early days Network and System Management

8. Emerging technologies in enterprise computing (Gartner Hype Cycle, Summer 2009)

9. What’s new? The analog: moving from files to DBMS In recent years – architectures, abstractions, and dedicated commercial products emerge to support functionality that was traditionally carried out within regular programming. For some applications it is an improvement in TCO; for others is breaking the cost-effectiveness barrier.

10. What is an event There are various definitions of events Event is an occurrence within a particular system or domain; the word event has double meaning: the real-world Occurrence as well as its computerized representation

11. In daily life we often react to events..

12. Sometime we even react to the occurrence of multiple events I closed the deal with the Canadians We closed two huge deals in a single day, It is a good opportunity to send all the team to Some fun time in Singapore I closed the deal with the Australians

13. Event Patterns Pattern detection is one of the notable functions of event processing

14. What we actually want to react to are – situations TOLL VILOATOR FRUSTRATED CUSTOMER Sometimes the situation is determined by detecting that some pattern occurred in the Flowing events. Toll violation Frustrated customer Sometimes the events can approximate or indicate with some certainty that the situation has occurred

15. Event processing is being used for various reasons

16. Event Driven Architecture Event driven architecture: asynchronous, decoupled; each component is autonomic.

17. Ancestor: active databases On event When condition Do action With coupling mode Composite events were inherited to event processing

18. Ancestor: Data Stream management system Source: Ankur Jain’s website

19. Ancestor: Temporal databases There is a substantial temporal nature to event processing. Recently – also spatial and spatio-temporal functions are being added

20. Ancestor: Messaging – pub/sub middleware

21. Ancestor: Discrete event simulation

22. Ancestor: Network and system management

23. Event processing – architecture, building blocks

24. Fast Flower Delivery Flower Store Van Driver Ranking and Reporting System Bid Request Delivery Bid Assignments, Bid alerts, Assign Alerts Control System GPS Location Location Service Location Driver’s Guild Ranking and reports Delivery confirmation Pick Up confirmation Ranked drivers / automatic assignment Bid System Store Preferences Delivery Request Assignment System Manual Assignment Assignment Assignments, Pick Up Alert Delivery Alert http://www.ep-ts.com/EventProcessingInAction

25. Event Processing Agent Context Event Channel Event Consumer Event Type Event Producer Global State The seven Building blocks

26. Event processing network

27. Example of EPN – part of the FFD example

28. Event type definition Detection time, Occurrence time, source, Certainty… Stock id, quote, volume… Free comments…

29. Producer – State Observer in workflows State observer Push: Instrumentation points; Pull: Query the state

30. Producer – Code instrumentation

31. Producer – syndication

32. Producers – video streams to events

33. Producer – sensors

34. Producer and consumer - Sixth sense

35. Twitter as a producer and consumer

36. Consumer - Performance monitoring dashboard

37. Consumer - Ambient Orb

38. Event Processing Agent Filter Transform Detect Pattern Translate Aggregate Split Compose Enrich Project Event Processing Agents

39. The EPA picture

40. Filter EPA A filter EPA is an EPA that performs filtering only, and has no matching or derivation steps, so it does not transform the input event .

41. Transform EPA sub types

42. Sample of pattern types all pattern is satisfied when the relevant event set contains at least one instance of each event type in the participant set any pattern is satisfied if the relevant event set contains an instance of any of the event types in the participant set absence pattern is satisfied when there are no relevant events relative N highest values pattern is satisfied by the events which have the N highest value of a specific attribute over all the relevant events, where N is an argument value average pattern is satisfied when the value of a specific attribute, averaged over all the relevant events, satisfies the value average threshold assertion. always pattern is satisfied when all the relevant events satisfy the always pattern assertion sequence pattern is satisfied when the relevant event set contains at least one event instance for each event type in the participant set, and the order of the event instances is identical to the order of the event types in the participant set. increasing pattern is satisfied by an attribute A if for all the relevant events, e1 << e2  e1.A < e2.A relative max distance pattern is satisfied when the maximal distance between any two relevant events satisfies the max threshold assertion moving toward pattern is satisfied when for any pair of relevant events e1, e2 we have e1 << e2  the location of e2 is closer to a certain object then the location of e1.

43. Pattern detection example Pattern name: Manual Assignment Preparation Pattern Type: relative N highest Context: Bid Interval Relevant event types: Delivery Bid Pattern parameter: N = 5; value = Ranking Cardinality: Single deferred Find the five highest bids within the bid interval Taken from the Fast Flower Delivery use case

44. Why do we need policies - A simple example: heavy trading scenario Given: A stream of events of a single topic, about the activity in the stock market for a certain stock. An event is produced every 10 minutes when there is trade in the stock. Each event consists of: quote (current stock-quote), volume (an accumulated volume of traded events within these 10 minutes). The pattern based action is: trigger an automatic trade program if the volume exceeds 300,000 3 times within an hour; pass as an argument the last quote and the sum of the 3 volume values. How many times the trade programming is triggered ; Which arguments are used in each triggering? Why defining patterns is not that easy? Because we need to tune up the semantics

45. Pattern policies Evaluation policy —This determines when the matching sets are produced Deferred Cardinality policy —This determines how many matching sets are produced within a single context partition Repeated type policy —This determines what happens if the matching step encounters multiple events of the same type Every - Override – First – Last – with max/min value of…

46. Pattern policies – cont. Consumption policy —This specifies what happens to a participant event after it has been included in a matching set Consume Reuse Bounded reuse Order policy —This specifies how temporal order is defined By occurrence time – by detection time – by stream position – by attribute

47. Our entire culture is context sensitive In the play “The Tea house of the August Moon” one of the characters says: Pornography question of geography This says that in different geographical contexts people view things differently Furthermore, the syntax of the language (no verbs) is typical to the way that the people of Okinawa are talking When hearing concert people are not talking, eating, and keep their mobile phone on “silent”.

48. Context has three distinct roles (which may be combined) Partition the incoming events The events that relate to each customer are processed separately Grouping events together Different processing for Different context partitions Determining the processing Grouping together events that happened in the same hour at the same location

49. Context Definition A context is a named specification of conditions that groups event instances so that they can be processed in a related way. It assigns each event instance to one or more context partitions . A context may have one or more context dimensions. Temporal Spatial State Oriented Segmentation Oriented

50. Context Types Examples Spatial State Oriented Temporal Context “ Every day between 08:00 and 10:00 AM” “ A week after borrowing a disk” “ A time window bounded by TradingDayStart and TradingDayEnd events” “ 3 miles from the traffic accident location” “ Within an authorized zone in a manufactory” “ All Children 2-5 years old” “ All platinum customers” “ Airport security level is red” “ Weather is stormy” Segmentation Oriented

51. Context Types Context Fixed location Entity distance location Event distance location Spatial State Oriented Fixed interval Event interval Sliding fixed interval Sliding event interval Temporal Segmentation Oriented

52. The present: State of the practice in event processing – languages, implementation issues, challenges in implementing event processing applications

53. An Observation The Babylon Tower symbolizes the tendency Of humanity to talk in multiple languages. The Event Processing area is no different: most languages in the industry really follow the hammer and nails syndrome – and extended existing approaches imperative script language SQL extensions Extension of inference rule language The epts language analysis workgroup is aimed to understand the various styles And extract common functions that can be used to define what is an event processing language; this tutorial is an interim report It does not seem that we’ll succeed to settle In the near future around a single programming style

54. Existing Styles for EP languages (samples) Inference Rules State oriented Agent Oriented Imperative/ Script Based RuleCore Oracle Aleri Streambase Esper EventZero TIBCO WBE Apama Spade AMiT Netcool Impact * - if we add simple and mediated event processing the picture is even more diversified ECA Rules SQL extension Coral8 Agent Logic Starview XChange EQ Prova

55. StreamBase Studio

56. StreamBase Pattern Matching

57. CCL Studio (Coral8  Sybase)

58. CCL – Pattern Matching RFID monitoring application Checks if a tag has been seen by readers A and B, then C, but not D, within a 10 second window. Insert into StreamAlerts Select StreamA.id From StreamA a, StreamB b, StreamC c, StreamD d Matching [10 seconds: a && b, c, !d] On a.id = b.id = c.id = d.id

59. Microsoft Streaminsights var topfive = (from window in inputStream.Snapshot() from e in window orderby e.f ascending, e.i descending select e).Take(5); var avgCount = from v in inputStream group v by v.i % 4 into eachGroup from window in eachGroup.Snapshot() select new { avgNumber = window.Avg(e => e.number) };

60. Esper EPL – FFD Example /* * Not delivered up after 10 mins (600 secs) of the request target delivery time */ insert into AlertW(requestId, message, driver, timestamp) select a.requestId, &quot;not delivered&quot;, a.driver, current_timestamp() from pattern[ every a=Assignment  (timer:interval(600 + (a.deliveryTime-current_timestamp)/1000) and not DeliveryConfirmation(requestId = a.requestId) and not NoOneToReceiveMSG(requestId = a.requestId)) ];

61. ruleCore - Reakt Event stream view - a unique context of events a view contains a window into the inbound stream of events and contains commonly only semantically related events Situation - an interesting combination of multiple events as they occur over time An item with an RFID tag being picked up from the shelf and then moving past the checkout without being paid for Rule - an active event processing entity reacting to specific combinations of inbound events over time Action - the last part of a rule's evaluation in response to a detected situation

62. Amit Terminology Situation Conditions Lifespan e5 e8 Initiator Terminator Event Selection Actions Operation Keys e1 e3 Input events Operator Joining Counting Temporal Absence Aggregation Notifications Messages Definition updates User plug-ins e2

63. Amit - Situation

64. IBM Websphere Business Events

65. Apama EPL – FFD Examples

66. Apama Simulation Studio – cont.

67. Non functional properties - scalability scalability in the volume of processed events Scalability in the quantity of agents Scalability in the quantity of producers scalability in the quantity of consumers scalability in the quantity of context partitions Scalability in context state size Scalability in the complexity of computation scalability in the processor environment Scalability is the capability of a system to adapt readily to a greater or lesser intensity of use, volume , or demand while still meeting its business objectives .

68. Non functional requirements - security Ensuring only authorized parties are allowed to be event producers of event consumers Ensuring that incoming events are filtered so that authorized producers cannot introduce invalid events, or events that they are not entitled to publish Ensuring that consumers only receive information to which they are entitled. In some cases a consumer might be entitled to see some of the attributes of an event but not others. Ensuring that unauthorized parties cannot add new EPAs to the system, or make modifications to the EPN itself (in systems where dynamic EPN modification is supported) Keeping auditable logs of events received and processed, or other activities performed by the system. Ensuring that all databases and data communications links used by the system are secure.

69. Performance objectives

70. Optimizations Optimizations related to EPA assignment: partition, parallelism, Distribution, load balancing. Optimization related to the execution process: scheduling, routing optimizations and load shedding Optimizations related to the coding of specific EPAs: code optimization, state management.

71. Putting derived events in order

72. Ordering in a distributed environment - possible issues The occurrence time of an event is accurate, but the event arrives out-of-order and processing that should have included the event might already been executed. Neither the occurrence time nor detection time can be trusted, so the order of events cannot be accurately determined.

73. Solutions: Buffering techniques: Assumptions: Events are reported by the producers as soon as they occur; The delay in reporting events to the system is relatively small, and can be bounded by a time-out offset; Events arriving after this time-out can be ignored. Principles: Let  be the time-out offset, according to the assumption it is safe to assume that at any time-point t, all events whose occurrence time is earlier than t -  have already arrived. Each event whose occurrence time is To is then kept in the buffer until To+  , at which time the buffer can be sorted by occurrence time, and then events can be processed in this sorted order. Retrospective compensation: Find out all EPAs that have already sent derived events which would have been affected by the &quot;out-of-order&quot; event if it had arrived at the right time. Retract all the derived events that should not have been emitted in their current form. Replay the original events with the late one inserted in its correct place in the sequence so that the correct derived events are generated.

74. Inexact event processing uncertainty whether an event actually occurred Uncertainty when exactly the event happened inexact content in the event payload inexact matching between derived events and the situations they purport to describe

75. The future: Trends and research challenges

76. Trend I: Going from narrow to wide Border security radiation detection Mobile asset geofence Logistic and scheduling Unauthorized use of heavy machinery Hospital patient and asset tracking Activity monitoring for taxing and fraud detection Intelligent CRM in banking EDA and asynchronous BPM in retail Situation awareness in energy utilities Situation awareness in airlines Reduce cost in injection therapy Next generation navigation Real-time management of hazardous materials Finding anomalies in point of sales in retail stores Elderly behavior monitoring Some recently reported applications (EPTS use-cases WG) Source: ebizQ Event processing market pulse

77. Trend I: Going from narrow to wide Taking event processing outside enterprise computing: Robotics Bio-Informatics Socio-technical systems Home Automation

78. Trend II: Going from monolithic to diversified Variety of functions Variety of Quality of Service requirements: Variety of platforms “ One size fits all” will not work – Instead a collection of building blocks that can fit together

79. Trend III: Going from proprietary to standard-based – standard directions The current situation: Babylon tower: variety of languages, event representation… Serves as enabler to achieving other trends and general maturity The shift of vendors from start-up dominant to bigger companies makes the atmosphere more friendly towards standards. Areas for Standards: Modeling Event representation Interoperability Languages PIM

80. Trend IV: Going from programmer centered to semi-technical person centered Source: ebizQ Event processing market pulse

81. Trend V: Going from stand-alone to embedded Packaged applications Business Activity Monitoring Sensor Platform Middleware and platforms

82. Trend VI: Going from reactive to proactive

83. Emerging directions: Four directions to observe Multiple platforms – same look and feel The engineering of constructing EP applications Adding intelligence to Event processing Tailor-made optimizations

84. Emerging direction I: multiple platforms – same look and feel

85. Emerging direction II: Tailor-made optimizations Local optimizations: each EPA will be optimized for its own purpose / assumptions / QoS indicators – average/worst case latency, input/output throughput … Global optimization: scheduling, load balancing, assignment… EPA EPA EPA EPA Producer Producer EPA Consumer Consumer Local optimizations Global optimizations

86. Emerging direction III: Event processing software engineering Best practices Methodologies Modeling & meta-modeling Design Patterns

87. Emerging direction IV: Intelligent event processing Offline and continuous mining of meaningful patterns in event histories Inexact event processing – handling inexact events and also false positives and false negatives Causality – a key for proactive, but also vital for provenance

88. Summary Event processing has emerged from multiple academic disciplines The state of the Practice is the 1 st generation of products – mainly engineering based Going to the next phase – many challenges that require collaboration of research and industry Already attracted coverage of analysts and all major software vendors The Dagstuhl grand challenge