Business Event Procesing Beyond The Horizon


Published on

This is a presentation given in IBM Websphere IMPACT 2009, May 2009, Las Vegas together with Kyle Brown. It contains some thoughts that are demonstrated through customers' scenarios on future functionality in event processing products.

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Business Event Procesing Beyond The Horizon

  1. 1. Business Event Processing – Beyond the Horizon Kyle Brown, Opher Etzion
  2. 2. Our Vision: Event Processing in 2019 Event Processing repeats (in 30-something years offset) the success of “Data Management” Part of main stream computing Wide coverage in term of applications that are doing some type of event processing Broadly accepted standards Event Processing extensions to programming languages Large amount of developers are familiar with the concepts Widely taught in universities with popular textbooks Well-established Research community that contribute to the concepts and the engineering aspects Other disciplines focused on extracting events and event patterns (image processing, information retrieval, search engines, data mining).
  3. 3. An 2009 View In recent Years Event Processing has become one of the fastest growing segments of enterprise integration middleware There have been many talks in this conference about IBM’s Products in this space, and it was Mentioned a lot as a key enabler of the smarter planet However, event processing as a discipline is still in the relatively early phases; many more developments to this technology are expected beyond the horizon
  4. 4. Main challenges $ ! quot; # $ $
  5. 5. Platform Oriented Challenges Move from engines to platforms. Each platform can host a variety of specialized agents optimized for a specific task. The same platform will be embeddable in various higher level platforms – as event processing is typically a part of a b i g g e r p i c t u r e ' ' # ( )& # ( )& % $$ & % & % % quot; quot;
  6. 6. Engineering Oriented Challenges QoS Driven , * quot; + $ quot; quot; EPN EPN Event Event EPA Producer EPA Consumer EPA Pattern Event Event Producer Consumer EPA EPA Pattern Event Event Pattern Consumer Producer
  7. 7. User Oriented Challenges ) $ + ' ,$ / + / / - ' .
  8. 8. Functional challenges – the focus of our talk Automatically generating events and patterns Geo-Spatial Event Processing Processing the past and the future Uncertain event processing
  9. 9. How is this presentation structured? Kyle Brown, IBM Distinguished Dr. Opher Etzion, IBM Senior Engineer in IBM Software Services Technical Staff Member, and Support for Websphere will Event Processing Scientific focus on use cases to explain the Leader in IBM Research will Technologies focus on the technology side
  10. 10. Automatically generating events and patterns
  11. 11. Background: Events An Event is something that happens. Eureka Event representation in a computerized system answers questions like: • What happened ? • When did it happen ? • Where did it happen ? • Who was involved ? • What other information is relevant to understand this event? Current event processing systems process events that are typically structured and obtained by instrumentation (e.g. state observers), sensors (e.g. RFID tag readers), and adapters from various sources
  12. 12. More event sources Using various techniques Video Streams (such as: image Processing, voice analysis, information retrieval, natural language Processing) to understand the event and its details: What happened ? Audio Streams When did it happen ? Where did it happen ? Who was involved ? What other information is relevant to understand this event? Internet goodies
  13. 13. Examples: Extracting events from multi-media streams Many toll roads and traffic lights use video cameras to take pictures of license plates as a car passes by Allows the picture and license plate # to be extracted and used for billing or ticketing Can also extract sounds from a continuous audio stream Used by law enforcement to detect gunshots and determine both time and location
  14. 14. Examples: Extracting events from texts There have been many examples of people using Twitter feeds to represent events: NYU’s Botanicals group has made it possible for your plants to tweet you when they need watering One West coast Krispy Kreme donut franchise tweets their “Hot Donuts Now” sign
  15. 15. Background: Event Patterns One of the main function of business event Processing is “pattern matching”: find if a Certain combination of events happened. For example: • Find if the same customer Pattern Matching already made product inquiry about the same product recently (see below) • Find if a customer issued three complains already recently Event Processing The result of a pattern detection may be interpreted as “situation” – an occurrence in the user’s domain that requires notification / reaction
  16. 16. Extracting patterns from higher level abstractions In some cases the patterns can be extracted from legal documents, regulations, policies In other cases the pattern can be extracted from decision modeling The idea is to enable automated creation of patterns and in general the business logic behind BEP, to enable agility and reduce the long IT life-cycles.
  17. 17. Extracting patterns by machine learning techniques In some cases the patterns to be watched can be obtained by looking at past event and determine causalities among events using machine learning techniques. This can be static (off-line) or dynamic (on-line) learning.
  18. 18. Example: Automatic extraction of pattern and business logic • Analyze the event flows in money inflow and outflow and in declared investment strategies in hedge funds or mutual funds shown to be Ponzi schemes (like Madoff’s investment fund) • The stated Madoff investment strategy, called quot;split-strike conversion,quot; is known to be very volatile; it involves trading huge positions around options expirations. • Despite that, the fund’s returns over the past decade were a stable 8-10 percent. • These patterns can then be applied to existing money flows and detect Ponzi schemes currently in progress
  19. 19. Geo-Spatial Event Processing
  20. 20. Geo-spatial events and patterns GPS and other location sensors enable locating events and moving objects ! quot;! # quot; quot;! # quot; Patterns can be based on locations, for example: ! quot;! # quot; $ observe traffic patterns on highway, and quot;! # quot; track individual moving objects, ! quot;! # quot; ' $ quot;! # quot; % &
  21. 21. Geospatial Customer Examples Healthcare Event Processing: Tracking of medical equipment within a hospital campus – knowing that certain equipment needs to be within a certain room at a certain time Manufacturing Event Processing Tracking the arrival of parts into a work station Tracking the creation of parts as they are created Tracking the disposition of shared resources in a factory (such as pallets or forklifts) Shipping and Tracking event processing When does something arrive at a freight terminal When does the same object move on to the next stage in its journey
  22. 22. Processing the past and future
  23. 23. Retrospective Event Processing Situation Reinforcement: An event pattern designates the possibility that a business situation has occurred; in order to provide positive or negative reinforcement, as part of the on-line pattern detection, there is a need to find complementary pattern (which is typically not traced) in order to assert or refute the occurrence of the situation. Patterns for observations on past events Event Patterns can be used to find periodic observations about past events
  24. 24. Predictive Event Processing Processing events that have not yet happened: Event are predicted by causality relationships with other events or using predictive analysis tools Alerting, mitigating, adaption or eliminating the occurrence of the predictive events Alerts, and in some cases autonomous actions to decide how to mitigate past events.
  25. 25. Retrospective Customer Example I • on-line situation:. Money – A person that has deposited (in aggregate) more Laundering than $20,000 within a single working day is a SUSPECT in money laundering • Reinforcement situation (conjunction of…) – There has been a period of week within the last year in which the same person has deposited (in aggregate) $50,000 or more and has withdrawn (in aggregate) at least $50,000 within the same week. – The same person has already been a quot;suspectquot; according to this definition within the last 30 business days. • If the on-line situation occurs then look for the reinforcement situation – if it satisfied then the event quot;confirmed suspectquot; is derived.
  26. 26. Retrospective Customer Example II • An electronic trade site provides the opportunity to customers to offer items for sale, but letting them conduct a bid, and provide bid management system The Greedy Seller (using a CEP system, of course). One of the services it provides to the customer is quot;alert that you are over- estimating the price you can get” • On-line Situation: – There has been at least two bidders, however none of them have matched the minimum price of the seller then this may be an indication of quot;too expensive bidquot;. • Reinforcement Situation: – at least 2/3 of the past bids of the same sellers have also resulted in a quot;too expensive bidquot; situation, – If the on-line situation occurs then look for the reinforcement situation – if it satisfied then the event then send the seller a notification quot;you are too greedyquot;.
  27. 27. Predictive Customer Examples Simple transportation example: • Departure of a large number of rail cars from a shipping port is always followed up within 12 hours by arrival of a large (but smaller) number of rail cars at a major routing depot • This physically indicates the arrival of one or more container ships that have been unloaded and the containers shipped out • By analyzing this recurring traffic pattern the rail company could plan to reschedule track maintenance activities to reduce congestion
  28. 28. Uncertain Event Processing
  29. 29. Uncertain Events Uncertainty IF the event happened Uncertainty WHEN the event happened Uncertainty about the event content (exactly WHAT happened) Using techniques for representing and process uncertain Information, and adapt them to event processing
  30. 30. Uncertain Situations and Event Patterns Recall: Situation is something that requires reaction from the user’s point of view: it can be either a raw event, or a result of a detected pattern. The event or pattern may just approximate the conditions where the situation occurs, but may not have a complete match – example: a collection of medical symptoms may indicate a differential diagnosis (with some certainty measurement). Using techniques for uncertainty inference and reasoning, such as: fuzzy reasoning, Bayesian networks, evidential reasoning…
  31. 31. Customer Uncertainty Example • The traffic jam example: – Consider a truck freight routing system that takes as one input reports of traffic jams – If manual data entry is required then an event (such as a report of a traffic backup) may be reported within an uncertainty of several minutes – the backup may be cleared by the time it is reported – Also people may misreport a traffic jam (was that accident at the Corner of Main and 5th or Main and 4th?) – Likewise if the reporting of an event requires individual judgment then the existence of the event itself may be in doubt (what determines if it is a “Major” traffic backup)
  32. 32. Pattern Uncertainty example In many cases the pattern itself has an uncertainty figure attached to the result Example: Diagnostics rules – A diagnosis may be within a level of uncertainty (e.g. an 80% chance patient has a staph infection) This also applies to events derived from multimedia streams (e.g. handwriting or character recognition, voice recognition)
  33. 33. Summary The area of Event Processing just scratched the surface of its potential and is spreading to different directions, all based on customer applications we already identified in the present
  34. 34. Summary (II) IBM Research is actively involved in driving the IBM products in the Business Event Processing space to advance beyond the current state of the art IBM is also leading the event processing community to form the “Event Processing Technical Society” which is engaged in a community effort to advance the 2019 vision
  35. 35. We love your Feedback! • Don’t forget to submit your Impact session and speaker feedback! Your feedback is very important to us, we use it to improve our conference for you next year. • Go to on a smartphone device or a loaner device • From the Impact 2009 Online Conference Guide; – Select Agenda – Navigate to the session you want to give feedback on – Select the session or speaker feedback links – Submit your feedback
  36. 36. © IBM Corporation 2009. All Rights Reserved. The workshops, sessions and materials have been prepared by IBM or the session speakers and reflect their own views. They are provided for informational purposes only, and are neither intended to, nor shall have the effect of being, legal or other guidance or advice to any participant. While efforts were made to verify the completeness and accuracy of the information contained in this presentation, it is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this presentation or any other materials. Nothing contained in this presentation is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here. All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. The following are trademarks of the International Business Machines Corporation in the United States and/or other countries:, CICS, CICSPlex, DataPower, DB2, DB2 Universal Database, i5/OS, IBM, the IBM logo, IMS/ESA, Power Systems, Lotus, OMEGAMON, OS/390, Parallel Sysplex, pureXML, Rational, Redbooks, Sametime, SMART SOA, System z , Tivoli, WebSphere, and z/OS. A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries. IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency which is now part of the Office of Government Commerce Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office Intel and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.