Your SlideShare is downloading. ×
Debs 2013 tutorial : Why is event-driven thinking different from traditional thinking
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Debs 2013 tutorial : Why is event-driven thinking different from traditional thinking

1,531
views

Published on

Published in: Technology, Business

1 Comment
5 Likes
Statistics
Notes
  • Hi is there any voice of this presentation? it is amazing ...
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
1,531
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
106
Comments
1
Likes
5
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Why is event-driven thinking different from traditional thinking about computing? Presenters: Opher Etzion and Jeffrey Adkins
  • 2. A year ago, Roy Schulte from Gartner published a personal blog entitled “does anybody care about event processing” He admitted that his predictions about the actual size of the event processing market is smaller than predicted His observation is that 95% of the event processing market is not visible since it is home- built and not labeled as EP In this tutorial we will discuss what is event driven thinking and how it is possible to help organizations to help themselves in exploiting events
  • 3. © 2013, IBM Corporation Agenda I Introduction – Brief History of Event Processing in practice II The major differentiation factor of event- based thinking III The Ontology of event and event influence IV Anatomy of reactive systems V Pragmatics : A business oriented approach VI Summary 6/30/2013 3
  • 4. Topic I – Introduction & a brief history of event processing in practice
  • 5. © 2013, IBM Corporation Event Processing History First start-ups Descendents of academic projects Apama acquisition By Progress Around 2000 2007 TIBCO and Oracle announce products Streambase Coral8 2005 EPTS Established Hitting the analysts hype… 2008 IBM Joins 2012 New players: SAS. Yahoo, Twitter 2013 EP at the height of BIG DATA hype Cycle M&A: TIBCO/Streambase Software AG/Apama 6/30/2013 5
  • 6. © 2013, IBM Corporation Events, as “data in motion” is one of the fundamental ingredients in big data: event -driven analytics Event -driven services / making events part of SOA Event-based decision making and event driven Optimization Event -driven processing as a backbone of next generation systems: event-based robotic, autonomic vehicles, human enhancement technology… Where are event used today? Virtually everywhere 6/30/2013 6
  • 7. © 2013, IBM Corporation Event Processing in 2013 There is now an accelerated development of new event-based systems – many of the current trends in computing are event-driven Internet of Things 6/30/2013 7
  • 8. © 2013, IBM Corporation Relatively new players in event and stream processing Storm S4 Google Intelligent events IFTTT ON {X} 6/30/2013 8
  • 9. © 2013, IBM Corporation Big Data Hype Cycle 2012 Source: Gartner publication G00235042, July 31, 2012 6/30/2013 9 Event processing Again in the hype Cycle – in different context
  • 10. © 2013, IBM Corporation I want to know about it immediately and react in the best possible way Detect Derive DoDecide Awareness ReactionSituation Event Driven Applications follow the 4D paradigm 6/30/2013 10
  • 11. © 2013, IBM Corporation This is how the event-driven application market looks This is how the current event processing market looks like… Source: Event processing Manifesto manual Build your Own Use COTS New segments Source: Event processing Manifesto 6/30/2013 11
  • 12. © 2013, IBM Corporation EPMM – Event Processing Maturity Model 0: unused 2: implicit 3: islands 4: integrated 1: manual 5: strategic No event awareness Subscription to some events, manual handling Events are stored in databases and are processed as part of process oriented Explicit event processing for some applications as islands; instrumentation and actions are hard coded and sporadic Event processing is integrated with main business processes; instrumentation and actuators are well established Strategic view of event processing across the enterprise 6/30/2013 12
  • 13. © 2013, IBM Corporation Lack of standards: SOA took off only when WS standards were accepted Lack of sufficient awareness and good ROI understanding: Need entry points and methodology about benefits to individuals, enterprises, packaged applications providers. Luck of understanding of what is event-based thinking and how to translate it to implementation Lack of skills – current tools require highly skilled developers to do tricky programming 6/30/2013 13 Barriers for Wider Adoption
  • 14. © 2013, IBM Corporation Major Gap: Products in this area are geared towards IT Developers A comprehensive user survey shows that 84% of the users wish that event rules could be defined by business users There is a gap Current models: Implementation oriented Business analysts oriented Modeling 6/30/2013 14
  • 15. © 2013, IBM Corporation What makes it difficult to express requirements? Develop correct application with the right semantics Observation: A substantial amount of effort is invested today in many of the tools to workaround the inability of the language to easily create correct solutions 6/30/2013 15
  • 16. © 2013, IBM Corporation Some Correctness Topics The right interpretation of language constructs The right order of events The right classification of events to windows 6/30/2013 16
  • 17. © 2013, IBM Corporation A simple scenario to demonstrate complexity – why “native implementation” does not work? Bid scenario- ground rules: 1. All bidders that issued a bid within the validity interval participate in the bid. 2. The highest bid wins. In the case of tie between bids, the first accepted bid wins the auction Race conditions: Between events; Between events and Window start/end 6/30/2013 17
  • 18. © 2013, IBM Corporation A simple scenario to demonstrate complexity – why “native implementation” does not work? Bid scenario- ground rules: 1. All bidders that issued a bid within the validity interval participate in the bid. 2. The highest bid wins. In the case of tie between bids, the first accepted bid wins the auction ===Input Bids=== Bid Start 12:55:00 credit bid id=2,occurrence time=12:55:32,price=4 cash bid id=29,occurrence time=12:55:33,price=4 cash bid id=33,occurrence time=12:55:34,price=3 credit bid id=66,occurrence time=12:55:36,price=4 credit bid id=56,occurrence time=12:55:59,price=5 Bid End 12:56:00 ===Winning Bid=== cash bid id=29,occurrence time=12:55:33,price=4 Trace: Race conditions: Between events; Between events and Window start/end 6/30/2013 18
  • 19. © 2013, IBM Corporation Positively looking: what is the main challenge to resolve in order to accelerate the use of events? 6/30/2013 19 Advancing the technology is always helpful - it is happening, yet it is not the main challenge. Organizations are not interested in technology for technology’s sake alone The next frontier: change the thinking start with the business need and then get to the IT side.
  • 20. © 2013, IBM Corporation Summary of topic I: In the business level – it is not well understood how to think in events and how to utilize events In the application level – the life-cycle of event-based systems require skilled IT developers Next - we explain what is different about event-driven thinking 6/30/2013 20
  • 21. Topic II – the major differentiation of event-based thinking
  • 22. © 2013, IBM Corporation Deposit Transfer abroad A simple example: event-based Anti money Laundering Based on events: Identify suspicious AccountsAn account is suspicious if any of the following patterns are satisfied: 1. Frequent big cash deposit 2. Frequent cases of a big cash deposit followed by transfer abroad 3. Lack of account activity 4. Increasing amounts of deposits 6/30/2013 22 Compliance officer
  • 23. © 2013, IBM Corporation Characteristics of Event-Driven Scenarios Events trigger action Events influence logic for the results There may be multiple events whose combined content influences the results Temporal contexts (15 days) influence the results I want to know about it immediately and react in the best possible way A suspicious account is detected whenever there are at least three big cash deposits followed by transfers abroad in the last 15 days The next section provides an introduction to event processing 6/30/2013 23
  • 24. © 2013, IBM Corporation Traditional Thinking Insert the event into a database; use periodic or on- demand queries to process the events The processing may not be efficient – many of the requests will not yield results The processing may not be effective – the time to react may be missed 6/30/2013 24
  • 25. © 2013, IBM Corporation Process Oriented Thinking (EPMM Level 2) 6/30/2013 25
  • 26. © 2013, IBM Corporation Difficulties in expressing such scenario in traditional thinking The event-driven vs. request-driven nature Effectiveness and Efficiency issues The temporal oriented behavior The hidden state handling 6/30/2013 26
  • 27. © 2013, IBM Corporation Efficiency and effectiveness issues The processing may not be efficient – many of the requests will not yield results The processing may not be effective – the time to react may be missed 6/30/2013 27
  • 28. © 2013, IBM Corporation Request driven vs. event driven thinking 6/30/2013 28
  • 29. © 2013, IBM Corporation In daily life we often react to events.. 6/30/2013 29
  • 30. © 2013, IBM Corporation Traditionally programmers are trained to think in a request driven way Searching the web, database queries, use of web services, use of mobile applications 6/30/2013 30
  • 31. © 2013, IBM Corporation What are the differences in thinking? 6/30/2013 31 Question Response Driven Event Drive Why is an action being taken? As a response to a specific request Triggered by the fact of a specific situation When is an action being taken? When the request is being processed Determined based on the context of the situation What happens when the request / event occurs? A response is always produced The event can be ignored, increment the state, trigger an internal derive event, or trigger a situation
  • 32. © 2013, IBM Corporation Temporal consideration changes everything The logic is sensitive to timing of events A delivery should be confirmed by the deadline The logic is sensitive to the order of events The winner in the bid is the first one who made the highest bid Determination by timing considerations Driver ranking increase and decrease are determined every 20 assignments What?Why? When? 6/30/2013 32
  • 33. © 2013, IBM Corporation Temporal consideration changes everything The logic is sensitive to timing of events A delivery should be confirmed by the deadline The logic is sensitive to the order of events The winner in the bid is the first one who made the highest bid Determination by timing considerations Driver ranking increase and decrease are determined every 20 assignments What?Why? When? In traditional models temporal functions are hand-coded, adding complexity 6/30/2013 33
  • 34. © 2013, IBM Corporation Events during rush hour are of interest, events outside rush hour are not Events that occur or don’t occur relative to a deadline Is the reported problem already solved, or is it still open? 6/30/2013 34 Logic sensitive to the timing of events’ occurrences
  • 35. © 2013, IBM Corporation Who arrived first? Has the bid arrived while the auction was still open? 6/30/2013 35 Logic sensitive to the order of events’ occurrences
  • 36. © 2013, IBM Corporation Determine the status of a patient based on blood pressure measurements: Every 8 measurements Every 5 hours 6/30/2013 36 The nature of situation is determined by timing considerations
  • 37. © 2013, IBM Corporation Event Patterns Pattern “event1 occurs after event2” requires keeping state of all unmatched instances of event1 Handling Hidden State 6/30/2013 37
  • 38. © 2013, IBM Corporation Summary of topic II: In many cases – event driven functionality is expressed using the traditional request-response fashion Fundamental differences exist between the two paradigms, and benefits exist in using event-driven modeling and implementation for certain applications Next – drilling down to the essence of event driven thinking 6/30/2013 38
  • 39. Topic III – The ontology of events and event influence
  • 40. © 2013, IBM Corporation What is an event – three views An event is anything that happens, or is contemplated as happening. The happening view The state change view An event is a state of change of anything The detectable condition view An event is a detectable condition that can trigger a notification 6/30/2013 40
  • 41. © 2013, IBM Corporation Events in Linguistics thinking “Friends, you and me... you brought another friend... and then there were three...” 6/30/2013 41 Events that we want to know, can, and should, be first worked through as done as a sentence.
  • 42. © 2013, IBM Corporation Events in Linguistics thinking 6/30/2013 42 Consider the story and write it down. This captures the essence of the event.
  • 43. © 2013, IBM Corporation43 Ancient Criterion of Change 6/29/2013 An Object, x, changes if and only if i. 𝑡ℎ𝑒𝑟𝑒 𝑖𝑠 𝑎 𝑝𝑟𝑜𝑝𝑒𝑟𝑡𝑦, 𝑃, ii. 𝑡ℎ𝑒𝑟𝑒 𝑖𝑠 𝑎𝑛 𝑜𝑏𝑗𝑒𝑐𝑡, 𝑥, iii. 𝑡ℎ𝑒𝑟𝑒 𝑎𝑟𝑒 𝑑𝑖𝑠𝑡𝑖𝑛𝑐𝑡 𝑡𝑖𝑚𝑒𝑠, 𝑡 𝑎𝑛𝑑 𝑡’ 𝑡 ≠ 𝑡’ , 𝑎𝑛𝑑 iv. that 𝑥 ℎ𝑎𝑠 𝑃 𝑎𝑡 𝑡 𝑎𝑛𝑑 𝑓𝑎𝑖𝑙𝑠 𝑡𝑜 ℎ𝑎𝑣𝑒 𝑃 𝑎𝑡 𝑡’ (Lombard)
  • 44. © 2013, IBM Corporation44 Quality Space 6/29/2013 Quality Spaces are sets (S) of simple, static properties {P1, P2, … Pn} which meet conditions: (i) If at any time, t, any object, x, has 𝑃𝑖 ∈ 𝑆 then (ii) If at any time, t’, x doesn’t have 𝑃𝑖, 𝑖𝑡 𝑤𝑖𝑙𝑙 ℎ𝑎𝑣𝑒 𝑃𝑗 ∈ 𝑆 𝑤ℎ𝑒𝑟𝑒 𝑖 ≠ 𝑗 We call this a Dimension. An object can have multiple dimensions. Could be represented by an ERD, enumeration
  • 45. © 2013, IBM Corporation The Goal It is the MOVEMENT along this quality space that constitutes an event. The Goal: To become aware of these events, so that we may react to them. 6/30/2013 45
  • 46. © 2013, IBM Corporation Awareness Boundary What knowledge is accessible Awareness Boundary The Awareness boundary represents the boundary of an ecosystem’s knowledge of events / situations. This is the knowledge that an ecosystem uses to understand situations, decide on a course of action and perform that course of action. 6/30/2013 46
  • 47. © 2013, IBM Corporation Awareness Boundary Situation occur out side awareness Situation occur out side awareness Situation occur out side awareness Situation occur outside of awareness There are situations that occur outside of an ecosystem’s awareness boundary to which a reaction inside the ecosystem is warranted. 6/30/2013 47
  • 48. © 2013, IBM Corporation Awareness Boundary Situation occur out side awareness Situation occur out side awareness Situation occur out side awareness Situation occur outside of awareness Sensors Some situations’ occurrences can be directly detected by sensors or come into an ecosystem through data feeds or some other instrumentation. 6/30/2013 48
  • 49. © 2013, IBM Corporation Awareness Boundary Situation occur out side awareness Situation occur out side awareness Situation occur out side awareness Situation occur outside of awareness Sensors System is now aware. Once detected, a virtual representation of the situation, called an event, exists within the ecosystem. These event can be used as part of the work of the ecosystem. 6/30/2013 49
  • 50. © 2013, IBM Corporation Awareness Boundary Situation occur out side awareness Situation occur out side awareness Situation occur out side awareness Situation occur outside of awareness Sensors Because we can’t directly detect it we must derive it Other situations that occur outside our awareness boundary can’t be directly detected, so we have to derive that it occurred. 6/30/2013 50
  • 51. © 2013, IBM Corporation Awareness Boundary Situation occur out side awareness Situation occur out side awareness Situation occur out side awareness Situation occur outside of awareness Because we can’t directly detect it we derive it These two situations are indicators that the third situation has occurred and since we have knowledge of their occurrence, we can use it to derive the third, non-detectable situation. Indicators Indicators 6/30/2013 51 Indicated
  • 52. © 2013, IBM Corporation Awareness Boundary Situation occur out side awareness Situation occur out side awareness Situation occur out side awareness Situation occur outside of awareness Derive Mechanism Derive Because we can’t directly detect it we derive it Propose calling relationship between real world event and derived awareness of event a Luckham Relationship There is a relationship between the real world situation’s occurrence and the virtual representation. We propose calling it “a Luckham relationship”. Indicators Indicators 6/30/2013 52 Indicated
  • 53. © 2013, IBM Corporation Awareness Boundary Situation occur out side awareness Situation occur out side awareness Situation occur out side awareness Situation occur outside of awareness Derive Mechanism Derive There are still other situations that cannot be directly detected nor derived. These situations require a human to observe it and enter it in the system. Indicators Indicators 6/30/2013 53 Indicated
  • 54. © 2013, IBM Corporation Awareness Boundary Situation occur out side awareness Situation occur out side awareness Situation occur out side awareness Situation occur outside of awareness Derive Mechanism Derive This is actually the most common way situations become known. These too should generate an event inside the ecosystem to separate out awareness from reaction. 6/30/2013 54
  • 55. © 2013, IBM Corporation Luckham Relationship These two situations, when occurred together in a pattern, indicated that the situation on top has occurred. Situation of Concern 6/30/2013 55
  • 56. © 2013, IBM Corporation Luckham Relationship There are different criteria of change that may play a part in this pattern indicating the situation on top occurred. Situation of Concern 6/30/2013 56
  • 57. © 2013, IBM Corporation Luckham Relationship This pattern also has a probability associated with it which indicates the confidence that the situation on top occurred. Situation of Concern 6/30/2013 57
  • 58. © 2013, IBM Corporation Luckham Relationship Situation of Concern , , 6/30/2013 58 A particular situation may have more than one pattern that indicates to a certain level of confidence that it has occurred.
  • 59. © 2013, IBM Corporation Object – Nouns of our story 6/30/2013 59 Nouns The things we manipulate, collect, buy, sell, talk with…. It has state and dimensions It has relations with other THINGS
  • 60. © 2013, IBM Corporation All participates in the process. • Actor (hero) is the one who performs the change • Helper assists “hero” • Obstacle inhibits the process. These can help tell about what is happening. Actors – The Hero, The Helper, and The Obstacle 6/30/2013 60
  • 61. © 2013, IBM Corporation Observer – Reports what is seen 6/30/2013 61 Observer is not part of the process There are potentially multiple observers of the same events Observer might have a subjective inaccurate perspective Observers may not be familiar with the context
  • 62. © 2013, IBM Corporation Processes Some Noun of Importance This noun is something we care about 6/30/2013 62 We have some noun / thing that is important to the business and we want to know when it changes.
  • 63. © 2013, IBM Corporation Processes Some Noun of Importance but different A process is needed to alter the noun The noun is altered Process 6/30/2013 63 Some Noun of Importance This noun is something we care about A process consumes that noun and transforms it into something more useful. Transforms
  • 64. © 2013, IBM Corporation Processes Some Noun of Importance but different A process is needed to alter the noun The noun is altered Process 6/30/2013 64 Some Noun of Importance This noun is something we care about When a noun is changed, it gives of indicators that can be sensed. Transforms Gives off indicators of the change
  • 65. © 2013, IBM Corporation Processes Some Noun of Importance but different A process is needed to alter the noun The altered noun. Process 6/30/2013 65 Some Noun of Importance This noun is something we care about An event is created inside the ecosystem that is the logical representation of indicators Transforms Gives off indicators of the change Event “noun has changed” The event is a logical representation of the indicators
  • 66. © 2013, IBM Corporation Process Ownership 666/29/2013 Nature •Seemly Random but patterns emerge •Impact small to momentous/ catastrophic Regulatory •Due Process •Rules by Law & Procedure •Impact significant Competitor •Little visibility until in open market •Actively hiding info •Share Market Space Supply Chain Partner •Visibility to a point •Vested interest in mutual success •Other Customers Self •Own Process •Easy instrumentation •Have Understanding Processes understood And available indicators Processes hidden And opaque indicators Process can be owned by one of the five group below. Which have several characteristics. The spectrum of each shows how much of the process and indicators are available or obscured.
  • 67. © 2013, IBM Corporation Data POV vs. Process POV 676/29/2013 Nature Regulatory Competitor Supply Chain Partner Own Processes Understood And Available indicators Processes hidden And opaque indicates Instrumentation Sensors, Market Intelligence Collectors, Industry Process, Object self-publish, transactions, feeds Tooling Streaming, Big Data, Master data management Processing / Transaction flow D a t a P O V P r o c e s s P O V When processes and indicators are understood and available, we take a Process POV approach. When they are hidden and opaque, we take a data oriented approach.
  • 68. © 2013, IBM Corporation Awareness Engineer 686/29/2013 Nature Regulatory Competitor Supply Chain Partner Own Starting Point 1. Identifying situations, streams / datasets available 2. Determine patterns that indicate situation 3. Iterate over intermediate patterns until you get to raw data/events 1. Identifying process triggers in terms of situation 2. Identify significant nouns, its states and properties 3. Map situation to noun-state change 4. Instrument in processes code significant state changes (detect) 5. For all situations that does not directly map, iterate top-down or bottoms-up until you connect (derive) Data POV Process POV The Awareness Engineer job is to figure out how we become aware. Below are starting points for the awareness engineer based on the Point of View.
  • 69. Awareness of events – what are the main reasons? Business IT Consumers 6/30/2013 69
  • 70. Reaction types Decision first: A decision is needed in order to make reaction; the decision can be simple, or complex (requiring OR methods) Actuators: Automatic activation of actuator Notification in various ways: Activation of process/workflow/task – manual or automatic 6/30/2013 70
  • 71. © 2013, IBM Corporation By 2015, 80% of all available data will be uncertain GlobalDataVolumeinExabytes Multiple sources: IDC,Cisco 100 90 80 70 60 50 40 30 20 10 AggregateUncertainty% 9000 8000 7000 6000 5000 4000 3000 2000 1000 0 2005 2010 2015 Data quality solutions exist for enterprise data like customer, product, and address data, but this is only a fraction of the total enterprise data. By 2015 the number of networked devices will be double the entire global population. All sensor data has uncertainty. The total number of social media accounts exceeds the entire global population. This data is highly uncertain in both its expression and content. 6/30/2013 71
  • 72. © 2013, IBM Corporation Representative sources of uncertainty Uncertain input data/ Events Source Malfunction Thermometer Human error Malicious Source Fake tweet Sensor disrupter Projection of temporal anomalies Wrong hourly sales summary Source Inaccuracy Sampling or approximation Propagation of uncertainty Visual data Rumor Wrong trend Inference based on uncertain value 6/30/2013 72
  • 73. © 2013, IBM Corporation Types of uncertainty in event processing Runtime Engine Definitions Detected Situations Event Sources Run Time Build Time Events Rules / Patterns Situation Detection Authoring Tool Actions incomplete event streams insufficient event dictionary erroneous event recognition inconsistent event annotation imprecise event patterns Uncertainty in the event input, in the composite event pattern, in both 6/30/2013 73
  • 74. © 2013, IBM Corporation Uncertainty handling Two main handling methods: Uncertainty propagation The uncertainty of input events is propagated to the derived events Uncertainty flattening Uncertain values are replaced with deterministic equivalents; events may be ignored. Traditional event processing needs to be enhanced to account for uncertain events 6/30/2013 74
  • 75. © 2013, IBM Corporation Patternmatching: Sequence(1/3) Crime report matching Pattern: Sequence [Suspicious observation, Crime report] Context: Location, Crime type Certainty 0.8 Occurrence time Uni(9:45AM,10:05AM) Id ‘John Doe’ ….. ...... Suspicious Observation Certainty 0.9 Occurrence time 10:02AM Id NA ….. ...... Crime report 6/30/2013 75
  • 76. © 2013, IBM Corporation Crime report matching Pattern: Sequence Context: Location, Crime type Certainty 0.8 Occurrence time Uni(9:45AM,10:05AM) Id ‘John Doe’ ….. ...... Suspicious Observation Certainty 0.9 Occurrence time 10:02AM Id NA ….. ...... Crime report Certainty 0.612 Occurrence time Uni(9:45AM,10:02AM) Id ‘John Doe’ ….. ...... Matched crime obs.certainty * crime.certainty * Prob{obs.time<crime.time} obs.time | obs.time<crime.time The ‘uncertainty propagation’ approach Pattern matching: Sequence (2/3) 6/30/2013 76
  • 77. © 2013, IBM Corporation Crime report matching Pattern: Sequence Context: Location, Crime type Certainty 0.8 Occurrence time Uni(9:45AM,10:05AM) Id ‘John Doe’ ….. ...... Suspicious Observation Certainty 0.9 Occurrence time 10:02AM Id NA ….. ...... Crime report Certainty 0.72 Occurrence time 9:55AM Id ‘John Doe’ ….. ...... Matched crime Occurrence time → percentile(occurrence time, 0.5) The ‘uncertainty flattening’ approach Pattern matching: Sequence (3/3) 6/30/2013 77
  • 78. © 2013, IBM Corporation Summary of topic III: The ontology of event-based systems is based on awareness of events – either directly or by derivation from other events The awareness is enabler for reactions Next - we describe the anatomy of event-based systems 6/30/2013 78
  • 79. Topic IV – Anatomy of reactive systems
  • 80. © 2013, IBM Corporation Two separate but connected goals: Awareness and Reaction Awareness Reaction Event Detect Derive Decide Do 6/30/2013 80
  • 81. © 2013, IBM Corporation Reactions are Events too Becoming aware of an event and then doing something about it. Derive Mechanisms Single Event Multiple Events Ancillary Info May need multiple iterations May require addition reference / state information Something we want to react to A Situation Detect Mechanism Feedback Events Decide Mechanism Do Mechanism Event of Interest Order Should be indicating Entity State Change Should be indicating Decisions Derived Event Trigger 6/30/2013 81
  • 82. © 2013, IBM Corporation Detect Some Noun of Importance but different The act of bringing into a system’s sphere of understanding knowledge about an event. A person recognizes the change and enters it into some system. This is the classic case! This is the most flexible because humans are ingenious. 6/30/2013 82
  • 83. © 2013, IBM Corporation Detect Some Noun of Importance but different The act of bringing into a system’s sphere of understanding knowledge about an event. A sensor senses the indicators and creates the corresponding event in the system. 6/30/2013 83
  • 84. © 2013, IBM Corporation Detect Some Noun of Importance but different The act of bringing into a system’s sphere of understanding knowledge about an event. A data feed or systemic interface allows events to be published into the system. 6/30/2013 84
  • 85. © 2013, IBM Corporation Detect Some Noun of Importance but different The act of bringing into a system’s sphere of understanding knowledge about an event. SwimLane TriggerEvent Activity State Change When the processes under the system’s control makes changes to nouns it should published these changes as events. 6/30/2013 85
  • 86. © 2013, IBM Corporation Detect Some Noun of Importance but different The act of bringing into a system’s sphere of understanding knowledge about an event. SwimLane TriggerEvent Activity State Change As connected things are becoming more self-aware of their inner- workings, they can publish their own state changes. 6/30/2013 86
  • 87. © 2013, IBM Corporation Derive The act of becoming aware of events that are not directly detectable by bringing together events with other events, data, patterns and publishing the observation as a derived event. Raw events Raw events Raw events A Person recognizes the pattern and enters the derived event or just reacts to it directly. Shown a lot of time by dashboards and analysis. 6/30/2013 87
  • 88. © 2013, IBM Corporation Derive The act of becoming aware of events that are not directly detectable by bringing together events with other events, data, patterns and publishing the observation as a derived event. Raw events Raw events Raw events A Neural network processes the various inputs and determines a new situation expressed by a derived event. 6/30/2013 88
  • 89. © 2013, IBM Corporation Derive The act of becoming aware of events that are not directly detectable by bringing together events with other events, data, patterns and publishing the observation as a derived event. Raw events Raw events Raw events A software applies pattern matching over multiple events and data to find derived events. 6/30/2013 89
  • 90. © 2013, IBM Corporation Derive The act of becoming aware of events that are not directly detectable by bringing together events with other events, data, patterns and publishing the observation as a derived event. Raw events Raw events Raw events The most common place is hidden inside of every day system’s code. 6/30/2013 90
  • 91. © 2013, IBM Corporation Decide The act of determining the course of action to do in response to the situation. This includes the background information needed to be collected to make the decision. Pass through: Sometimes there is no decision. There is only one course of action. 6/30/2013 91
  • 92. © 2013, IBM Corporation Decide The act of determining the course of action to do in response to the situation. This includes the background information needed to be collected to make the decision. Manual Decision: Many times the ecosystem asks a person to decide the course of take. 6/30/2013 92
  • 93. © 2013, IBM Corporation Decide The act of determining the course of action to do in response to the situation. This includes the background information needed to be collected to make the decision. Automated Decision: Algorithmic decision via a decision management system. 6/30/2013 93
  • 94. © 2013, IBM Corporation Decide The act of determining the course of action to do in response to the situation. This includes the background information needed to be collected to make the decision. Automated Goal Oriented: Algorithmic decision via a decision management system that seeks a optimizing quantitative goals. 6/30/2013 94
  • 95. © 2013, IBM Corporation Do The act of performing the course of action that was decided upon. Notification: Sending a signal of sort to either a person or system. This would include calling a web-service or subscription to alerts. 6/30/2013 95
  • 96. © 2013, IBM Corporation Do The act of performing the course of action that was decided upon. Manual Action: This is an order for a human to go do an action. 6/30/2013 96
  • 97. © 2013, IBM Corporation Do The act of performing the course of action that was decided upon. Applying Actuator: cause a action or setting change on an actuator. 6/30/2013 97
  • 98. © 2013, IBM Corporation Do The act of performing the course of action that was decided upon. Trigger process: Execute a process or potential a single action. 6/30/2013 98
  • 99. © 2013, IBM Corporation Richard Hackathorn’s Response Time Latency Value in terms of Competitiveness decreases 6/30/2013 99 Source: Richard Hackathorn – Active data warehouse, from nice to necessary, Teradata Magazine, 2006
  • 100. © 2013, IBM Corporation 4D Version of Response Time Latency Business Event Detected Derived Decide Started Decide Completed Do Started Do Completed Value Time Detect Latency Derive Latency Decide latency Do latency Queue ActQueue Act 6/30/2013 100 Converting Richard Hackathorn’s flow to the 4D Prospective.
  • 101. © 2013, IBM Corporation Detect Latency Business Event Detected Derived Decide Started Decide Completed Do Started Do Completed Value Time Detect Latency Derive Latency Decide latency Do latency Queue ActQueue Act 6/30/2013 101 The time it takes for a ecosystem to detect either the business event or indicators that can be used derive the event.
  • 102. © 2013, IBM Corporation Derive Latency 6/30/2013 102 Business Event Detected Derived Decide Started Decide Completed Do Started Do Completed Value Time Detect Latency Derive Latency Decide latency Do latency Queue ActQueue Act The time necessary to combine the indicators, events, ancillary data or human analysis to derive situations that can’t be directly detected.
  • 103. © 2013, IBM Corporation Decide Queue Latency 6/30/2013 103 Business Event Detected Derived Decide Started Decide Completed Do Started Do Completed Value Time Detect Latency Derive Latency Decide latency Do latency Queue ActQueue Act The time waiting for someone (manual decisions) or something (automated decisions) to make a decision.
  • 104. © 2013, IBM Corporation Decide Act Latency 6/30/2013 104 Business Event Detected Derived Decide Started Decide Completed Do Started Do Completed Value Time Detect Latency Derive Latency Decide latency Do latency Queue ActQueue Act The time it takes to decide the course of action in reaction to the situation.
  • 105. © 2013, IBM Corporation Do Queue Latency 6/30/2013 105 Business Event Detected Derived Decide Started Decide Completed Do Started Do Completed Value Time Detect Latency Derive Latency Decide latency Do latency Queue ActQueue Act The time waiting for someone (human actor) or something (machine actor) to start executing or orchestrating the course of action.
  • 106. © 2013, IBM Corporation Business Event Detected Derived Decide Started Decide Completed Do Started Do Completed Value Time Detect Latency Derive Latency Decide latency Do latency Queue ActQueue Act Do Act latency 6/30/2013 106 The time it takes to execute the course of action.
  • 107. © 2013, IBM Corporation 4D Version of Response Time Latency Value Investment Value Realization 6/30/2013 107 Until the course of action is completed, everything is a value investment. After the course of action is completed, value can be realized. Business Event Detected Derived Decide Started Decide Completed Do Started Do Completed Value Time Detect Latency Derive Latency Decide latency Do latency Queue ActQueue Act
  • 108. © 2013, IBM Corporation Business velocity – a key competitive edge 6/30/2013 108
  • 109. © 2013, IBM Corporation Summary of topic IV: An event-driven system consists of the 4D architecture Latency reduction is the key to business velocity Next we’ll present basic ideas about the business oriented approach 6/30/2013 109
  • 110. Topic V – pragmatics – a computational independent model for event-based systems
  • 111. © 2013, IBM Corporation ROI Business Goals Application characteristics Current state in the maturity model Approaching Event Processing in an Enterprise 6/30/2013 111
  • 112. © 2013, IBM Corporation Business Predictability Enable the support of event-driven adaptive business processes Business Agility Business Optimization Make faster and better (manual or autonomic) decisions based on timely multi-source information; ROI Business Goals Application characteristics operational support proactively seeking and adapting to patterns that might indicate an emerging event (threats and opportunities) Increasing level of automation and thus increase productivity (e.g. of back office) and reduce cost Continuous audit ensures timely handling of violations. Quantify the impact of: Compliance with Regulation Business Velocity Increase business velocity by faster response to opportunities and early detection of threats Calculating the benefits of event processing 6/30/2013 112
  • 113. © 2013, IBM Corporation Situation awareness Enables the business logic to be context sensitive Context sensitive Real-time dissemination Taking advantage of information whose value decreases in time. ROI Business Goals Application characteristics Fast change The application includes sense and respond to events, and situation awareness is key requirement Enables fast deployment of new versions when the business logic dynamically changes Situation determination complexity Complexity stems from one or more of: event rate, quantity of event sources, state and context handling, event order sensitivity.. Quantify the level of relevance to each candidate application: Return on Investment 6/30/2013 113
  • 114. © 2013, IBM Corporation EPMM: Event Processing Maturity Model (IT View) 0: unused 3: islands 4: integrated 1: manual 5: strategic No event awareness Subscription to some events, manual handling Events are stored in databases and are processed as part of process oriented Explicit event processing for some applications as islands ; instrumentation and actions are hard coded and sporadic Event processing is integrated with main business processes; instrumentation and actuators are well established Strategic view of event processing cross the enterprise 2: implicit 6/30/2013 114
  • 115. © 2013, IBM Corporation Next Step: Develop Multi-Step Method to Select Entry Points Improve Business Agility Improve Business Optimization Select Goals Prioritize Goals 1 Improve Business Agility 2 Improve Business Optimization Analyze the possible entry point Applications based on their characteristics Select applications that best fit these goals Produce plan of action Based on the maturity model By entry point, map both satisfaction of business goals and fitness characteristics for best ROI. Liquidity Management Anti Money Laundering . . . 6/30/2013 115
  • 116. © 2013, IBM Corporation Vision: Shift Governance from Programmer to Knowledge Worker Governance occurs through development and maintenance of program code. Governance occurs through development and maintenance of event models. TODAY TOMORROW CODE LEVEL 6/30/2013 116
  • 117. © 2013, IBM Corporation Process Oriented View 6/30/2013 117
  • 118. © 2013, IBM Corporation Next step – event processing language (ESPER) // Big cash deposit insert into BigCashDeposit select * from Transaction where amount > 100,000 and transaction_cash_deposit_indicator =’Y’ // Frequent (At least three) big cash deposits create context AccountID partition by accountId on Transaction; Context AccountID Insert into FrequentBigCashDeposits select count(*) from BigCashDeposit having count(*)>3; // Transfer abroad insert into TransferAbroad select * from Transaction where transferabroad_indicator =’Y’ // Frequent cash deposits followed by transfer abroad Context AccountID insert into SuspiciousAccount select * from pattern [ every f=FrequentCashDeposit -> t=TransferAbroad where timer.within(15 days)] 6/30/2013 118
  • 119. © 2013, IBM Corporation Modeling according to the concept computing principles The application logic should be expressed by a semantically declarative, directly executable, implementation independent, and rigorously structured knowledge model Knowledge Model Automatic translation to code in regular or specific engine language Free of implementation assumptions Rigorous verifiable structure with all connections Represented as a collection of tables The term was coined by Mills Davis in 2012 6/30/2013 119
  • 120. © 2013, IBM Corporation TEM Logic Specification Frequent big cash deposits Row # Context Event Logic When Partition By Regular Multiple events Conclusion Conditions Conditions Expression Start End Account ID Count(Big cash deposit) Frequent big cash deposits 1 last 15 days same > 3 is Derived 6/30/2013 120 Lack of account activity Row # Context Event Logic When Partition By Regular Multiple events Conclusion Conditions Conditions Expression Start End Account ID Transaction Lack of account activity 1 last 10 days same is Absent is Derived
  • 121. © 2013, IBM Corporation Next step – model view Big cash deposit ______________________________________ - - - - - - - - - - - - - - transaction cash deposit indicator transaction amount (customer threshold) ______________________________________ Always Customer ID Frequent big cash deposits ______________________________________ Big cash deposit - - - - - - - - - - - - - - ______________________________________ Last 10 days Account ID Frequent deposits followed by transfers abroad ______________________________________________ Deposit followed by transfer abroad - - - - - - - - - - - - - - _________________________________________ Last 30 days Account ID Lack of account activity ___________________________________________ - - - - - - - - - - - - - - Transaction is absent ___________________________________________ Last 10 days Account ID Suspicious account ________________________________________________ Frequent big cash deposits Frequent deposits followed by transfers abroad Lack of account activity - - - - - - - - - - - - - - ________________________________________________ Always Account ID Derive Suspicious account Deposit followed by transfer abroad ______________________________________ Big cash deposit - - - - - - - - - - - - - - Transfer abroad ___________________________________________ deposit occurrence Account ID Bank transaction system Transaction Transfer abroad Transaction Compliance officer 6/30/2013 121
  • 122. Concepts of Facts Actors Events States Event Derivation Logic Transitions GoalsIT elements Glossary Logic Computation Logic 6/30/2013 122
  • 123. © 2013, IBM Corporation Summary of topic V: Systematic approach in an enterprise based on ROI and maturity models -the business world is not interested in technology but in business outcomes Business user oriented modeling as a key solution point Next we’ll present basic ideas about computing independent model 6/30/2013 123
  • 124. Topic VI – Summary
  • 125. © 2013, IBM Corporation What is the main take away from this tutorial – 1/3? 6/30/2013 125 The exploitation of events is a game changer in the universe In the business level – it is not well understood how to think in events and how to utilize events In the IT level – most of what can be functionally thought as event processing is implemented using the traditional thinking within application code
  • 126. © 2013, IBM Corporation An event-driven system consists of the 4D architecture Latency reduction is the key to business velocity 6/30/2013 126 What is the main take away from this tutorial – 2/3? The ontology of event-based systems is based on awareness to events – either directly or by derivation from another events
  • 127. © 2013, IBM Corporation Systematic approach in an enterprise based on ROI and maturity models -the business world is not interested in technology but in business outcomes Business user oriented modeling as a key solution point 6/30/2013 127 What is the main take away from this tutorial – 3/3? Next step: help the organizations help themselves with realizing the benefits of events
  • 128. © 2013, IBM Corporation6/30/2013 128
  • 129. © 2013, IBM Corporation Presenters Opher Etzion Jeff Adkins IBM Research Haifa, Israel opher@il.ibm.com IBM GBS Washington, DC jmadkins@us.ibm.com Thanks for listening